</p> <p class="has-line-data" data-line-end="1" data-line-start="0">Cross validation is a widely used model validation approach. As we discussed in <a href="https://blogs.oracle.com/machinelearning/cross-validation-using-oml4py-part-i">Part I</a>, it has the benefit of providing a more complete picture of model performance. However, cross validation can be costly with large datasets since one needs to repeat the training and testing for K times if a K-fold cross validation is chosen.</p> <p class="has-line-data" data-line-end="4" data-line-start="3">By leveraging embedded Python execution in OML4Py, we can parallelize the K train/test processes, which is suitable for the use case when the user has an open source model to evaluate through cross validation. In this blog, we will show our approach to speed up cross validation for open source models using embedded Python execution.</p> <p class="has-line-data" data-line-end="11" data-line-start="8">We use the dataset customer insurance lifetime value for our demonstration, an Oracle-produced dataset. The use case involves an insurance company targeting customers likely to buy insurance based on their lifetime value, demographic, and financial features for each customer. The following is a glimpse into this dataset with a subset of the columns.</p> <p class="has-line-data" data-line-end="11" data-line-start="8"><img alt src="https://i2.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/7248b80d89b213a1427cf47aad11a17e/data1.png?w=1440&ssl=1" style="width: 950px; height: 236px;" data-recalc-dims="1" data-lazy-src="https://i2.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/7248b80d89b213a1427cf47aad11a17e/data1.png?w=1440&is-pending-load=1#038;ssl=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class=" jetpack-lazy-image"><noscript><img alt="" src="https://i2.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/7248b80d89b213a1427cf47aad11a17e/data1.png?w=1440&ssl=1" style="width: 950px; height: 236px;" data-recalc-dims="1"/></noscript><br /><img alt src="https://i1.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/9bec832e91ea08b713eee4285b9cc8cb/data2.png?w=1440&ssl=1" style="width: 833px; height: 241px;" data-recalc-dims="1" data-lazy-src="https://i1.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/9bec832e91ea08b713eee4285b9cc8cb/data2.png?w=1440&is-pending-load=1#038;ssl=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class=" jetpack-lazy-image"><noscript><img alt="" src="https://i1.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/9bec832e91ea08b713eee4285b9cc8cb/data2.png?w=1440&ssl=1" style="width: 833px; height: 241px;" data-recalc-dims="1"/></noscript></p> <p class="has-line-data" data-line-end="13" data-line-start="12">Based on the column names, we can see that the dataset contains demographic features of users such as state, region, gender, marital status, and some financial features like income, credit card limits.</p> <p class="has-line-data" data-line-end="16" data-line-start="15">The main business problem here is to find out which customers are likely to buy an insurance policy. From the dataset, the ground truth is that a customer has purchased the insurance policy based on the column BUY_INSURANCE. This is a typical binary classification problem and we can use all columns provided in this dataset to build the model.</p> <p class="has-line-data" data-line-end="19" data-line-start="18">We first divide the dataset into K= 10 parts using the KFold() function as we presented in Part I.</p> <pre> <code class="has-line-data" data-line-end="24" data-line-start="20">DF = oml.sync(table = <span class="hljs-string">'CUST_SUBSET_TBL10'</span>) fold = <span class="hljs-number">10</span> pairs = DF.KFold(n_splits = fold) </code></pre> <p class="has-line-data" data-line-end="26" data-line-start="25">We obtained a list of pairs of OML dataframes, which is in the form (train, test). Note here we did not materialize each pair of train and test dataset, rather they exist as proxy objects for views of underlying physical table. All we need is to pass those pairs of OML dataframes to the function doing cross validation. The function doing the cross validation will be described in the next section.</p> <p class="has-line-data" data-line-end="31" data-line-start="30">We use the following user-defined Python function to carry out the validation for each fold. Note that we also added code to measure the time elapsed for pulling the data, training and scoring. We also pass a function train_helper as the argument to this function so that the user can switch the algorithm easily.</p> <pre> <code class="has-line-data" data-line-end="81" data-line-start="32"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">build_open_src</span><span class="hljs-params">(idx, pairs, train_helper)</span>:</span> <span class="hljs-keyword">import</span> oml <span class="hljs-keyword">import</span> random <span class="hljs-keyword">import</span> time <span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd <span class="hljs-keyword">from</span> datetime <span class="hljs-keyword">import</span> datetime <span class="hljs-keyword">from</span> sklearn.ensemble <span class="hljs-keyword">import</span> RandomForestClassifier <span class="hljs-keyword">from</span> sklearn.ensemble <span class="hljs-keyword">import</span> GradientBoostingClassifier <span class="hljs-keyword">from</span> sklearn.metrics <span class="hljs-keyword">import</span> roc_auc_score time_df = pd.DataFrame(columns = [<span class="hljs-string">'START'</span>, <span class="hljs-string">'END'</span>, <span class="hljs-string">'DURATION'</span>, <span class="hljs-string">'JOB_TYPE'</span>, <span class="hljs-string">'JOB_ID'</span>]) <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">ts_now</span><span class="hljs-params">()</span>:</span> <span class="hljs-keyword">return</span> datetime.now().strftime(<span class="hljs-string">"%Y-%m-%d %H:%M:%S.%f"</span>) start_time = ts_now() start = time.perf_counter() train_dat = pairs[idx-<span class="hljs-number">1</span>][<span class="hljs-number">0</span>].pull() test_dat = pairs[idx-<span class="hljs-number">1</span>][<span class="hljs-number">1</span>].pull() end = time.perf_counter() end_time = ts_now() print(f<span class="hljs-string">'Pull() finished in {round(end -start, 2)} second(s) for job {idx}'</span>) time_df = time_df.append({<span class="hljs-string">'START'</span>: start_time, <span class="hljs-string">'END'</span>: end_time, <span class="hljs-string">'DURATION'</span>: end -start, <span class="hljs-string">'JOB_TYPE'</span>: <span class="hljs-string">'pull'</span>, <span class="hljs-string">'JOB_ID'</span>: idx}, ignore_index = <span class="hljs-keyword">True</span>) start_time = ts_now() start = time.perf_counter() X_train = train_dat[[<span class="hljs-string">'CREDIT_BALANCE'</span>, <span class="hljs-string">'MORTGAGE_AMOUNT'</span>, <span class="hljs-string">'BANK_FUNDS'</span>, <span class="hljs-string">'NUM_DEPENDENTS'</span>, <span class="hljs-string">'INCOME'</span>, <span class="hljs-string">'CREDIT_CARD_LIMITS'</span>]] y_train = train_dat[[<span class="hljs-string">'BUY_INSURANCE'</span>]] clf = train_helper(X_train, y_train) end = time.perf_counter() end_time = ts_now() print(f<span class="hljs-string">'Training finished in {round(end -start, 2)} second(s) for job {idx}'</span>) time_df = time_df.append({<span class="hljs-string">'START'</span>: start_time, <span class="hljs-string">'END'</span>: end_time, <span class="hljs-string">'DURATION'</span>: end -start, <span class="hljs-string">'JOB_TYPE'</span>: <span class="hljs-string">'train'</span>, <span class="hljs-string">'JOB_ID'</span>: idx}, ignore_index = <span class="hljs-keyword">True</span>) start_time = ts_now() start = time.perf_counter() X_test = test_dat[[<span class="hljs-string">'CREDIT_BALANCE'</span>, <span class="hljs-string">'MORTGAGE_AMOUNT'</span>, <span class="hljs-string">'BANK_FUNDS'</span>, <span class="hljs-string">'NUM_DEPENDENTS'</span>, <span class="hljs-string">'INCOME'</span>, <span class="hljs-string">'CREDIT_CARD_LIMITS'</span>]] y_test = test_dat[[<span class="hljs-string">'BUY_INSURANCE'</span>]] auc = roc_auc_score(y_test, clf.predict_proba(X_test)[:, <span class="hljs-number">1</span>]) end_time = ts_now() end = time.perf_counter() print(f<span class="hljs-string">'Scoring and metric finished in {round(end -start, 2)} second(s) for job {idx}'</span>) time_df = time_df.append({<span class="hljs-string">'START'</span>: start_time, <span class="hljs-string">'END'</span>: end_time, <span class="hljs-string">'DURATION'</span>: end -start, <span class="hljs-string">'JOB_TYPE'</span>: <span class="hljs-string">'score'</span>, <span class="hljs-string">'JOB_ID'</span>: idx}, ignore_index = <span class="hljs-keyword">True</span>) <span class="hljs-keyword">try</span>: oml.drop(table = <span class="hljs-string">'RECORD_DF_'</span> + str(idx)) <span class="hljs-keyword">except</span>: <span class="hljs-keyword">pass</span> _ = oml.create(time_df, table = <span class="hljs-string">'RECORD_DF_'</span> + str(idx)) <span class="hljs-keyword">return</span> auc </code></pre> <p class="has-line-data" data-line-end="82" data-line-start="81">Let us do a cross validation on sklearn random forest model, we write the function for training as below</p> <pre> <code class="has-line-data" data-line-end="89" data-line-start="83"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">train_rf</span><span class="hljs-params">(X_train, y_train)</span>:</span> <span class="hljs-keyword">from</span> sklearn.ensemble <span class="hljs-keyword">import</span> RandomForestClassifier clf = RandomForestClassifier(n_jobs = <span class="hljs-number">1</span>) clf.fit(X_train, y_train) <span class="hljs-keyword">return</span> clf </code></pre> <p class="has-line-data" data-line-end="90" data-line-start="89">We can write a loop to iterate over from idx = 1 to 10.</p> <pre> <code class="has-line-data" data-line-end="94" data-line-start="91"><span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">1</span>, <span class="hljs-number">11</span>): build_open_src(idx, pairs, train_helper = train_rf) </code></pre> <p class="has-line-data" data-line-end="95" data-line-start="94">The code prints the time spent in each stage. The following is a sample of the output.</p> <pre> <code class="has-line-data" data-line-end="104" data-line-start="96">Pull() finished <span class="hljs-keyword">in</span> <span class="hljs-number">0.4</span> second(s) <span class="hljs-keyword">for</span> job <span class="hljs-number">1</span> Training finished <span class="hljs-keyword">in</span> <span class="hljs-number">6.42</span> second(s) <span class="hljs-keyword">for</span> job <span class="hljs-number">1</span> Scoring <span class="hljs-keyword">and</span> metric finished <span class="hljs-keyword">in</span> <span class="hljs-number">0.01</span> second(s) <span class="hljs-keyword">for</span> job <span class="hljs-number">1</span> Pull() finished <span class="hljs-keyword">in</span> <span class="hljs-number">0.32</span> second(s) <span class="hljs-keyword">for</span> job <span class="hljs-number">2</span> Training finished <span class="hljs-keyword">in</span> <span class="hljs-number">6.0</span> second(s) <span class="hljs-keyword">for</span> job <span class="hljs-number">2</span> Scoring <span class="hljs-keyword">and</span> metric finished <span class="hljs-keyword">in</span> <span class="hljs-number">0.01</span> second(s) <span class="hljs-keyword">for</span> job <span class="hljs-number">2</span> ...</code></pre> <p class="has-line-data" data-line-end="105" data-line-start="104">Running a loop for K= 10 cross validation takes 10 times of the original time. Finishing the process takes around 40 seconds. If we repeated this process on a larger dataset with duplicated records up to 1 million rows, this process takes around 50 minutes. This is a significant delay in the model validation process. Remember in the daily job of a data scientist, every time a new feature or a new data cleaning method is applied, we need to run this cross validation process and this will run repeatedly, which incurs a significant time cost.</p> <p class="has-line-data" data-line-end="108" data-line-start="107">In the code shown above, we run the K-fold cross validation iteratively. We need to repeat the train and test process K times. If we can parallelize the process then we can save a lot of time.</p> <p class="has-line-data" data-line-end="111" data-line-start="110">To do this, we use OML4Py embedded Python execution, which allows the user to spawn multiple Python engines and run open source Python packages against data pulled from Oracle Database. By using the oml.index_apply function, we can start K Python engines running in the Oracle Autonomous Database (ADB) environment. Each Python engine runs one fold of training and testing as one job. We can obtain the auc scores altogether after the K jobs finish.</p> <p class="has-line-data" data-line-end="116" data-line-start="113">Let’s look at an example, the code below uses index_apply(), where the input argument to the user-defined Python function is the index of invocation (a value between 1 and the times specified), effectively, a job id.<br />Inside the function, the user is free to use the pre-installed open source package scikit-learn against the data pulled from ADB to the Python engine(s) automatically spawned by the database environment.<br />Then we can invoke index apply as follows.</p> <pre> <code class="has-line-data" data-line-end="119" data-line-start="117">res = oml.index_apply(times=<span class="hljs-number">10</span>, func=build_model, oml_connect=<span class="hljs-keyword">True</span>, parallel = <span class="hljs-number">8</span>, pairs = pairs, train_helper = train_rf) </code></pre> <p class="has-line-data" data-line-end="121" data-line-start="119">After this function finishes, all outputs of the AUC scores are accumulated into the output res. We can have a look at the distribution of the AUC using a boxplot</p> <p class="has-line-data" data-line-end="121" data-line-start="119"><img alt src="https://i0.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/cc723c810109f19400dd5f912f7dee2a/box.png?w=1440&ssl=1" style="width: 600px; height: 400px;" data-recalc-dims="1" data-lazy-src="https://i0.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/cc723c810109f19400dd5f912f7dee2a/box.png?w=1440&is-pending-load=1#038;ssl=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class=" jetpack-lazy-image"><noscript><img alt="" src="https://i0.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/cc723c810109f19400dd5f912f7dee2a/box.png?w=1440&ssl=1" style="width: 600px; height: 400px;" data-recalc-dims="1"/></noscript></p> <p class="has-line-data" data-line-end="121" data-line-start="119">Let’s look at each invocation of the user-defined Python function run inside these Python engines and plot the progress for each.<br />First, we collect and transform the data recorded in the cross validation function using the following two functions.</p> <pre> <code class="has-line-data" data-line-end="154" data-line-start="126"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">combine_plot</span><span class="hljs-params">(table_name, num)</span>:</span> RECORD_DF = oml.sync(table = table_name + <span class="hljs-string">'_1'</span>) <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">2</span>, num + <span class="hljs-number">1</span>): RECORD_DF = RECORD_DF.append(oml.sync(table = table_name + <span class="hljs-string">'_'</span> + str(i))) record_df = RECORD_DF.pull() record_df[<span class="hljs-string">'START'</span>] = pd.to_datetime(record_df[<span class="hljs-string">'START'</span>], format=<span class="hljs-string">"%Y-%m-%d %H:%M:%S.%f"</span>) record_df[<span class="hljs-string">'END'</span>] = pd.to_datetime(record_df[<span class="hljs-string">'END'</span>], format=<span class="hljs-string">"%Y-%m-%d %H:%M:%S.%f"</span>) <span class="hljs-keyword">return</span> record_df <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">plot_sub</span><span class="hljs-params">(record_df, job_types, ax, max_range)</span>:</span> min_start = min(record_df[<span class="hljs-string">'START'</span>].values) <span class="hljs-keyword">for</span> job_type <span class="hljs-keyword">in</span> job_types: plot_df = record_df[record_df[<span class="hljs-string">'JOB_TYPE'</span>] == job_type] start = np.array(plot_df[<span class="hljs-string">'START'</span>].values) end = np.array(plot_df[<span class="hljs-string">'END'</span>].values) start_s = (start - min_start) start_s = [s/ np.timedelta64(<span class="hljs-number">1</span>, <span class="hljs-string">'s'</span>) <span class="hljs-keyword">for</span> s <span class="hljs-keyword">in</span> start_s] duration = (end - start) duration = [d/ np.timedelta64(<span class="hljs-number">1</span>, <span class="hljs-string">'s'</span>) <span class="hljs-keyword">for</span> d <span class="hljs-keyword">in</span> duration] y = np.array(plot_df[<span class="hljs-string">'JOB_ID'</span>].values) y = [ str(s) <span class="hljs-keyword">for</span> s <span class="hljs-keyword">in</span> y] ax.barh(y, width = duration, left=start_s, label = job_type, color = colors[job_type]) ax.set_xlim(<span class="hljs-number">0</span>, max_range) ax.legend(ncol=len(job_types), bbox_to_anchor=(<span class="hljs-number">0</span>, <span class="hljs-number">1</span>), loc=<span class="hljs-string">'lower left'</span>, fontsize=<span class="hljs-string">'medium'</span>) ax.grid(axis=<span class="hljs-string">'x'</span>, color = <span class="hljs-string">'0.9'</span>) ax.set(xlabel=<span class="hljs-string">'Time (second) '</span>, ylabel=<span class="hljs-string">'Job ID'</span>) </code></pre> <p class="has-line-data" data-line-end="155" data-line-start="154">We run the parallel and serial version of the cross validation as follows。</p> <pre> <code class="has-line-data" data-line-end="162" data-line-start="156">res = oml.index_apply(times=<span class="hljs-number">10</span>, func=build_open_src, oml_connect=<span class="hljs-keyword">True</span>, parallel =<span class="hljs-number">8</span>, pairs = pairs, graphics = <span class="hljs-keyword">False</span>, train_helper = train_rf) index_df = combine_plot(<span class="hljs-string">'RECORD_DF'</span>, <span class="hljs-number">10</span>) <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">1</span>, <span class="hljs-number">11</span>): build_open_src(i, pairs, train_helper = train_rf) series_df = combine_plot(<span class="hljs-string">'RECORD_DF'</span>, <span class="hljs-number">10</span>) </code></pre> <p class="has-line-data" data-line-end="163" data-line-start="162">After that we are ready to generate the plot.</p> <pre> <code class="has-line-data" data-line-end="188" data-line-start="164"><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np <span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt colors = {<span class="hljs-string">'pull'</span>: <span class="hljs-string">'green'</span>, <span class="hljs-string">'train'</span>: <span class="hljs-string">'orange'</span>, <span class="hljs-string">'score'</span>: <span class="hljs-string">'cyan'</span>} fig, axs = plt.subplots(<span class="hljs-number">2</span>, figsize=(<span class="hljs-number">15</span>, <span class="hljs-number">10</span>)) job_types = [<span class="hljs-string">'pull'</span>, <span class="hljs-string">'train'</span>, <span class="hljs-string">'score'</span>] min_start = min(index_df[<span class="hljs-string">'START'</span>].values) max_end = max(index_df[<span class="hljs-string">'END'</span>].values) max_range = (max_end - min_start).item()/<span class="hljs-number">10</span>**<span class="hljs-number">9</span> min_start = min(series_df[<span class="hljs-string">'START'</span>].values) max_end = max(series_df[<span class="hljs-string">'END'</span>].values) max_range = max(max_range, (max_end - min_start).item()/<span class="hljs-number">10</span>**<span class="hljs-number">9</span>) plt.style.use(<span class="hljs-string">'seaborn-whitegrid'</span>) plot_sub(index_df, job_types, axs[<span class="hljs-number">0</span>], max_range) axs[<span class="hljs-number">0</span>].set_title(<span class="hljs-string">'Paralleled Run'</span>,fontweight=<span class="hljs-string">"bold"</span>, size=<span class="hljs-number">20</span>) upper = <span class="hljs-number">240</span> axs[<span class="hljs-number">0</span>].set_xlim([<span class="hljs-number">0</span>, upper]) major_ticks = np.arange(<span class="hljs-number">0</span>, upper, <span class="hljs-number">20</span>) axs[<span class="hljs-number">0</span>].set_xticks(major_ticks) plot_sub(series_df, job_types, axs[<span class="hljs-number">1</span>], max_range) axs[<span class="hljs-number">1</span>].set_title(<span class="hljs-string">'Serial Run'</span>,fontweight=<span class="hljs-string">"bold"</span>, size=<span class="hljs-number">20</span>) axs[<span class="hljs-number">1</span>].set_xticks(major_ticks) axs[<span class="hljs-number">1</span>].set_xlim([<span class="hljs-number">0</span>, upper]) plt.show() </code></pre> <p class="has-line-data" data-line-end="189" data-line-start="188">In the plot below, each horizontal row represents a single invocation. We use green to represent the time required to pull data from the database, orange as the time running the training job and cyan as the time spent in scoring.</p> <p class="has-line-data" data-line-end="191" data-line-start="190">The first subplot displays how things happened in the parallel setting. Under service level = MEDIUM, there are 4 Python engines spawned at the same time. The second subplot shows the serial runs, which is a sum of each job’s running time. We can see that in this case, we achieved more than 2.5 times of the performance gain.</p> <p class="has-line-data" data-line-end="191" data-line-start="190"><img alt src="https://i0.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/3932941d4cfffada587dd3b3b3cafb00/rf_medium_0727.png?w=1440&ssl=1" style="width: 1080px; height: 720px;" data-recalc-dims="1" data-lazy-src="https://i0.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/3932941d4cfffada587dd3b3b3cafb00/rf_medium_0727.png?w=1440&is-pending-load=1#038;ssl=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class=" jetpack-lazy-image"><noscript><img alt="" src="https://i0.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/3932941d4cfffada587dd3b3b3cafb00/rf_medium_0727.png?w=1440&ssl=1" style="width: 1080px; height: 720px;" data-recalc-dims="1"/></noscript></p> <p class="has-line-data" data-line-end="193" data-line-start="192">Let us repeat the experiment by using the HIGH service level. It shows that under HIGH, the OCPUs assigned to us is increased, so 8 jobs are started at the same time. This gives us a higher boost in performance (more than 3x).</p> <p class="has-line-data" data-line-end="193" data-line-start="192"><img alt src="https://i2.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/d879cd0441327d177fab378927a87ccd/croppped.png?w=1440&ssl=1" style="width: 1080px; height: 367px;" data-recalc-dims="1" data-lazy-src="https://i2.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/d879cd0441327d177fab378927a87ccd/croppped.png?w=1440&is-pending-load=1#038;ssl=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class=" jetpack-lazy-image"><noscript><img alt="" src="https://i2.wp.com/cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/00b40098-051d-415c-be23-4ceb933d5311/Image/d879cd0441327d177fab378927a87ccd/croppped.png?w=1440&ssl=1" style="width: 1080px; height: 367px;" data-recalc-dims="1"/></noscript></p> <p class="has-line-data" data-line-end="198" data-line-start="197">In this blog, we demonstrated the convenience and performance of running cross validation with OML4Py. By using .KFold function provided in the OML4Py transparency layer, cross validation is made easy. For users who want to use an open source python package, we provide a way to parallelize the cross validation jobs using index apply, and benefit from improved efficiency through embedded Python execution.</p> </p></div> <p><br /> <br /><a href="https://blogs.oracle.com/machinelearning/cross-validation-using-oml4py-part-ii"> Source link </a></p> <div class="post-views post-1708 entry-meta"> <span class="post-views-icon dashicons dashicons-chart-bar"></span> <span class="post-views-label">Post Views:</span> <span class="post-views-count">80</span> </div><!-- AddThis Advanced Settings above via filter on the_content --><!-- AddThis Advanced Settings below via filter on the_content --><!-- AddThis Advanced Settings generic via filter on the_content --><!-- AddThis Share Buttons above via filter on the_content --><!-- AddThis Share Buttons below via filter on the_content --><div class="at-below-post addthis_tool" data-url="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/"></div><!-- AddThis Share Buttons generic via filter on the_content --><div class="sharedaddy sd-sharing-enabled"><div class="robots-nocontent sd-block sd-social sd-social-official sd-sharing"><h3 class="sd-title">Share this:</h3><div class="sd-content"><ul><li class="share-twitter"><a href="https://twitter.com/share" class="twitter-share-button" data-url="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/" data-text="Cross Validation Using OML4Py Part II" data-via="sitworld" >Tweet</a></li><li class="share-facebook"><div class="fb-share-button" data-href="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/" data-layout="button_count"></div></li><li class="share-linkedin"><div class="linkedin_button"><script type="in/share" data-url="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/" data-counter="right"></script></div></li><li class="share-reddit"><div class="reddit_button"><iframe src="https://www.reddit.com/static/button/button1.html?newwindow=true&width=120&url=https%3A%2F%2Fmachinelearningmastery.in%2F2021%2F07%2F28%2Fcross-validation-using-oml4py-part-ii%2F&title=Cross%20Validation%20Using%20OML4Py%20Part%20II" height="22" width="120" scrolling="no" frameborder="0"></iframe></div></li><li class="share-telegram"><a rel="nofollow noopener noreferrer" data-shared="" class="share-telegram sd-button" href="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/?share=telegram" target="_blank" title="Click to share on Telegram"><span>Telegram</span></a></li><li class="share-jetpack-whatsapp"><a rel="nofollow noopener noreferrer" data-shared="" class="share-jetpack-whatsapp sd-button" href="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/?share=jetpack-whatsapp" target="_blank" title="Click to share on WhatsApp"><span>WhatsApp</span></a></li><li class="share-print"><a rel="nofollow noopener noreferrer" data-shared="" class="share-print sd-button" href="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/#print" target="_blank" title="Click to print"><span>Print</span></a></li><li class="share-tumblr"><a class="tumblr-share-button" target="_blank" href="https://www.tumblr.com/share" data-title="Cross Validation Using OML4Py Part II" data-content="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/" title="Share on Tumblr">Share on Tumblr</a></li><li class="share-pinterest"><div class="pinterest_button"><a href="https://www.pinterest.com/pin/create/button/?url=https%3A%2F%2Fmachinelearningmastery.in%2F2021%2F07%2F28%2Fcross-validation-using-oml4py-part-ii%2F&media=https%3A%2F%2Fi1.wp.com%2Fmachinelearningmastery.in%2Fwp-content%2Fuploads%2F2021%2F07%2Fdata1.png%3Ffit%3D950%252C236%26ssl%3D1&description=Cross%20Validation%20Using%20OML4Py%20Part%20II" data-pin-do="buttonPin" data-pin-config="beside"><img src="https://i2.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png?w=1440" data-recalc-dims="1" data-lazy-src="https://i2.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png?w=1440&is-pending-load=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class=" jetpack-lazy-image"><noscript><img src="https://i2.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png?w=1440" data-recalc-dims="1" /></noscript></a></div></li><li class="share-skype"><div class="skype-share" data-href="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/" data-lang="en-US" data-style="small" data-source="jetpack" ></div></li><li class="share-email"><a rel="nofollow noopener noreferrer" data-shared="" class="share-email sd-button" href="https://machinelearningmastery.in/2021/07/28/cross-validation-using-oml4py-part-ii/?share=email" target="_blank" title="Click to email this to a friend"><span>Email</span></a></li><li class="share-end"></li></ul></div></div></div><div class='sharedaddy sd-block sd-like jetpack-likes-widget-wrapper jetpack-likes-widget-unloaded' id='like-post-wrapper-170785677-1708-617025ea345db' data-src='https://widgets.wp.com/likes/#blog_id=170785677&post_id=1708&origin=machinelearningmastery.in&obj_id=170785677-1708-617025ea345db' data-name='like-post-frame-170785677-1708-617025ea345db'><h3 class="sd-title">Like this:</h3><div class='likes-widget-placeholder post-likes-widget-placeholder' style='height: 55px;'><span class='button'><span>Like</span></span> <span class="loading">Loading...</span></div><span class='sd-text-color'></span><a class='sd-link-color'></a></div> <div id='jp-relatedposts' class='jp-relatedposts' > <h3 class="jp-relatedposts-headline"><em>Related</em></h3> </div> </div> </div><!-- .entry-content --> <div class="screen-reader-text" itemprop="datePublished" itemtype="https://schema.org/Date">2021-07-28</div> </article><!-- .entry --> <div id="loop-nav-wrap" class="loop-nav"><div class="prev">Previous Post: <a href="https://machinelearningmastery.in/2021/07/28/an-ai-based-framework-solution-to-address-email-management-challenges/" rel="prev">An AI-Based Framework Solution to Address Email Management Challenges</a></div><div class="next">Next Post: <a href="https://machinelearningmastery.in/2021/07/29/a-brief-introduction-to-the-concept-of-data/" rel="next">A Brief Introduction to the Concept of Data</a></div></div><!-- .loop-nav --> <section id="comments-template"> <div id="respond" class="comment-respond"> <h3 id="reply-title" class="comment-reply-title">Leave a Reply <small><a rel="nofollow" id="cancel-comment-reply-link" href="/2021/07/28/cross-validation-using-oml4py-part-ii/#respond" style="display:none;">Cancel reply</a></small></h3><form action="https://machinelearningmastery.in/wp-comments-post.php" method="post" id="commentform" class="comment-form" novalidate><p class="comment-notes"><span id="email-notes">Your email address will not be published.</span></p><p class="comment-form-comment"><label for="comment">Comment</label> <textarea id="comment" name="comment" cols="45" rows="8" maxlength="65525" required="required"></textarea></p><p class="comment-form-author"><label for="author">Name</label> <input id="author" name="author" type="text" value="" size="30" maxlength="245" /></p> <p class="comment-form-email"><label for="email">Email</label> <input id="email" name="email" type="email" value="" size="30" maxlength="100" aria-describedby="email-notes" /></p> <p class="comment-form-url"><label for="url">Website</label> <input id="url" name="url" type="url" value="" size="30" maxlength="200" /></p> <p class="comment-form-cookies-consent"><input id="wp-comment-cookies-consent" name="wp-comment-cookies-consent" type="checkbox" value="yes" /> <label for="wp-comment-cookies-consent">Save my name, email, and website in this browser for the next time I comment.</label></p> <p class="wpgdprc-checkbox comment-form-wpgdprc"> <input type="checkbox" name="wpgdprc" id="wpgdprc" value="1" /> <label for="wpgdprc"> By using this form you agree with the storage and handling of your data by this website. <abbr class="wpgdprc-required" title="You need to accept this checkbox">*</abbr> </label> </p> <p class="form-submit"><input name="submit" type="submit" id="submit" class="submit" value="Post Comment" /> <input type='hidden' name='comment_post_ID' value='1708' id='comment_post_ID' /> <input type='hidden' name='comment_parent' id='comment_parent' value='0' /> </p><p style="display: none;"><input type="hidden" id="akismet_comment_nonce" name="akismet_comment_nonce" value="7d75c255d4" /></p><p style="display: none !important;"><label>Δ<textarea name="ak_hp_textarea" cols="45" rows="8" maxlength="100"></textarea></label><input type="hidden" id="ak_js" name="ak_js" value="79"/><script>document.getElementById( "ak_js" ).setAttribute( "value", ( new Date() ).getTime() );</script></p></form> </div><!-- #respond --> </section><!-- #comments-template --> </div><!-- #content-wrap --> </main><!-- #content --> <aside id="sidebar-primary" class="sidebar sidebar-primary hgrid-span-3 layout-narrow-right " role="complementary" itemscope="itemscope" itemtype="https://schema.org/WPSideBar"> <div class=" sidebar-wrap"> <section id="tag_cloud-3" class="widget widget_tag_cloud"><h3 class="widget-title"><span>Categories</span></h3><div class="tagcloud"><a href="https://machinelearningmastery.in/category/articles/" class="tag-cloud-link tag-link-404 tag-link-position-1" style="font-size: 11.400809716599pt;" aria-label="Articles (7 items)">Articles</a> <a href="https://machinelearningmastery.in/category/automation-anywhere/" class="tag-cloud-link tag-link-158 tag-link-position-2" style="font-size: 9.0202429149798pt;" aria-label="Automation Anywhere (2 items)">Automation Anywhere</a> <a href="https://machinelearningmastery.in/category/certification/" class="tag-cloud-link tag-link-12 tag-link-position-3" style="font-size: 10.267206477733pt;" aria-label="Certification (4 items)">Certification</a> <a href="https://machinelearningmastery.in/category/cloud/" class="tag-cloud-link tag-link-289 tag-link-position-4" style="font-size: 10.267206477733pt;" aria-label="Cloud (4 items)">Cloud</a> <a href="https://machinelearningmastery.in/category/code/" class="tag-cloud-link tag-link-511 tag-link-position-5" style="font-size: 8pt;" aria-label="Code (1 item)">Code</a> <a href="https://machinelearningmastery.in/category/database-2/" class="tag-cloud-link tag-link-593 tag-link-position-6" style="font-size: 8pt;" aria-label="Database (1 item)">Database</a> <a href="https://machinelearningmastery.in/category/data-science/" class="tag-cloud-link tag-link-9 tag-link-position-7" style="font-size: 11.967611336032pt;" aria-label="Data Science (9 items)">Data Science</a> <a href="https://machinelearningmastery.in/category/data-science-topics/" class="tag-cloud-link tag-link-530 tag-link-position-8" style="font-size: 9.0202429149798pt;" aria-label="data science topics (2 items)">data science topics</a> <a href="https://machinelearningmastery.in/category/data-science-update/" class="tag-cloud-link tag-link-13 tag-link-position-9" style="font-size: 22pt;" aria-label="Data Science Update (583 items)">Data Science Update</a> <a href="https://machinelearningmastery.in/category/deep-learning/" class="tag-cloud-link tag-link-290 tag-link-position-10" style="font-size: 11.117408906883pt;" aria-label="Deep Learning (6 items)">Deep Learning</a> <a href="https://machinelearningmastery.in/category/financial-assistance/" class="tag-cloud-link tag-link-8 tag-link-position-11" style="font-size: 8pt;" aria-label="Financial assistance (1 item)">Financial assistance</a> <a href="https://machinelearningmastery.in/category/google-cloud/" class="tag-cloud-link tag-link-583 tag-link-position-12" style="font-size: 9.0202429149798pt;" aria-label="Google Cloud (2 items)">Google Cloud</a> <a href="https://machinelearningmastery.in/category/interview-tips/" class="tag-cloud-link tag-link-181 tag-link-position-13" style="font-size: 8pt;" aria-label="Interview tips (1 item)">Interview tips</a> <a href="https://machinelearningmastery.in/category/machine-learning/" class="tag-cloud-link tag-link-11 tag-link-position-14" style="font-size: 15.368421052632pt;" aria-label="Machine Learning (39 items)">Machine Learning</a> <a href="https://machinelearningmastery.in/category/open-data-source/" class="tag-cloud-link tag-link-207 tag-link-position-15" style="font-size: 9.7004048582996pt;" aria-label="Open Data Source (3 items)">Open Data Source</a> <a href="https://machinelearningmastery.in/category/power-bi/" class="tag-cloud-link tag-link-341 tag-link-position-16" style="font-size: 9.0202429149798pt;" aria-label="Power BI (2 items)">Power BI</a> <a href="https://machinelearningmastery.in/category/project-management/" class="tag-cloud-link tag-link-409 tag-link-position-17" style="font-size: 9.7004048582996pt;" aria-label="Project Management (3 items)">Project Management</a> <a href="https://machinelearningmastery.in/category/python/" class="tag-cloud-link tag-link-2 tag-link-position-18" style="font-size: 11.967611336032pt;" aria-label="Python (9 items)">Python</a> <a href="https://machinelearningmastery.in/category/quiz-of-the-day/" class="tag-cloud-link tag-link-429 tag-link-position-19" style="font-size: 8pt;" aria-label="Quiz of the Day (1 item)">Quiz of the Day</a> <a href="https://machinelearningmastery.in/category/robotic-process-automation/" class="tag-cloud-link tag-link-159 tag-link-position-20" style="font-size: 9.0202429149798pt;" aria-label="Robotic Process Automation (2 items)">Robotic Process Automation</a> <a href="https://machinelearningmastery.in/category/r-programming/" class="tag-cloud-link tag-link-157 tag-link-position-21" style="font-size: 9.0202429149798pt;" aria-label="R Programming (2 items)">R Programming</a> <a href="https://machinelearningmastery.in/category/sas/" class="tag-cloud-link tag-link-156 tag-link-position-22" style="font-size: 10.720647773279pt;" aria-label="SAS (5 items)">SAS</a> <a href="https://machinelearningmastery.in/category/statistics/" class="tag-cloud-link tag-link-337 tag-link-position-23" style="font-size: 10.267206477733pt;" aria-label="Statistics (4 items)">Statistics</a> <a href="https://machinelearningmastery.in/category/tableau/" class="tag-cloud-link tag-link-340 tag-link-position-24" style="font-size: 9.0202429149798pt;" aria-label="Tableau (2 items)">Tableau</a> <a href="https://machinelearningmastery.in/category/visualization/" class="tag-cloud-link tag-link-10 tag-link-position-25" style="font-size: 12.194331983806pt;" aria-label="visualization (10 items)">visualization</a></div> </section><section id="newsletterwidget-2" class="widget widget_newsletterwidget"><div class="tnp tnp-widget"><form method="post" action="https://machinelearningmastery.in/?na=s"> <input type="hidden" name="nr" value="widget"><input type="hidden" name="nlang" value=""><div class="tnp-field tnp-field-email"><label for="tnp-email">Email</label> <input class="tnp-email" type="email" name="ne" value="" required></div> <div class="tnp-field tnp-field-button"><input class="tnp-submit" type="submit" value="Subscribe" > </div> </form> </div></section> <section id="recent-posts-2" class="widget widget_recent_entries"> <h3 class="widget-title"><span>Recent Posts</span></h3> <ul> <li> <a href="https://machinelearningmastery.in/2021/10/20/data-scientist-vs-data-engineer-salary/">Data Scientist vs Data Engineer Salary</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/19/choose-the-best-ai-accelerator-and-model-compilation-for-computer-vision-inference-with-amazon-sagemaker/">Choose the best AI accelerator and model compilation for computer vision inference with Amazon SageMaker</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/19/2021-data-engineer-salary-report-shares-insights-on-a-swiftly-evolving-market/">2021 Data Engineer Salary Report Shares Insights on a Swiftly Evolving Market</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/19/how-data-professionals-can-impress-even-when-busy/">How Data Professionals Can Impress Even When Busy</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/19/11-most-practical-data-science-skills-for-2022/">11 Most Practical Data Science Skills for 2022</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/19/how-to-create-an-interactive-dashboard-in-three-steps-with-knime-analytics-platform/">How to Create an Interactive Dashboard in Three Steps with KNIME Analytics Platform</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/18/top-stories-oct-11-17-query-your-pandas-dataframes-with-sql/">Top Stories, Oct 11-17: Query Your Pandas DataFrames with SQL</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/18/knowledge-graph-forum-technology-ecosystem-and-business-applications/">Knowledge Graph Forum: Technology Ecosystem and Business Applications</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/18/real-time-image-segmentation-using-5-lines-of-code/">Real Time Image Segmentation Using 5 Lines of Code</a> </li> <li> <a href="https://machinelearningmastery.in/2021/10/18/avoid-these-five-behaviors-that-make-you-look-like-a-data-novice/">Avoid These Five Behaviors That Make You Look Like A Data Novice</a> </li> </ul> </section> </div><!-- .sidebar-wrap --> </aside><!-- #sidebar-primary --> </div><!-- .main-content-grid --> </div><!-- #main --> <footer id="footer" class="site-footer footer hgrid-stretch inline-nav" role="contentinfo" itemscope="itemscope" itemtype="https://schema.org/WPFooter"> <div class="hgrid"> <div class="hgrid-span-6 footer-column"> <section id="hootkit-ticker-9" class="widget widget_hootkit-ticker"> <div class="ticker-widget ticker-usercontent ticker-simple ticker-userstyle ticker-withbg ticker-style1" style="background:#f1f1f1;color:#ff4530;" ><i class="fa-weixin fab ticker-icon"></i> <div class="ticker-msg-box" data-speed='0.03'> <div class="ticker-msgs"> <div class="ticker-msg"><div class="ticker-msg-inner">Subscribe for the latest news, updates, tips and more delivered right to your inbox.</div></div> </div> </div> </div></section> </div> <div class="hgrid-span-3 footer-column"> <section id="media_image-13" class="widget widget_media_image"><img width="220" height="49" src="https://i2.wp.com/machinelearningmastery.in/wp-content/uploads/2019/12/Machine-Learning-Mastery-banner.gif?fit=220%2C49&ssl=1" class="image wp-image-127 attachment-full size-full jetpack-lazy-image" alt="" loading="lazy" style="max-width: 100%; height: auto;" data-lazy-src="https://i2.wp.com/machinelearningmastery.in/wp-content/uploads/2019/12/Machine-Learning-Mastery-banner.gif?fit=220%2C49&ssl=1&is-pending-load=1" srcset="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" /></section> </div> <div class="hgrid-span-3 footer-column"> <section id="hootkit-social-icons-8" class="widget widget_hootkit-social-icons"> <div class="social-icons-widget social-icons-small"><a href="https://github.com/machinelearningmasteryindia" class=" social-icons-icon fa-github-block" target="_blank"> <i class="fa-github fab"></i> </a><a href="mailto:machinelearningmasteryindia@gmail.com" class=" social-icons-icon fa-envelope-block"> <i class="fa-envelope fas"></i> </a><a href="https://www.linkedin.com/in/machine-learning-b065081a9/" class=" social-icons-icon fa-linkedin-block" target="_blank"> <i class="fa-linkedin-in fab"></i> </a><a href="https://twitter.com/sitworld" class=" social-icons-icon fa-twitter-block" target="_blank"> <i class="fa-twitter fab"></i> </a></div></section> </div> </div> </footer><!-- #footer --> <div id="post-footer" class=" post-footer hgrid-stretch linkstyle"> <div class="hgrid"> <div class="hgrid-span-12"> <p class="credit small"> <a class="privacy-policy-link" href="https://machinelearningmastery.in/privacy-policy/">Privacy Policy</a> Designed using <a class="theme-link" href="https://wphoot.com/themes/unos/" title="Unos WordPress Theme">Unos</a>. Powered by <a class="wp-link" href="https://wordpress.org">WordPress</a>. </p><!-- .credit --> </div> </div> </div> </div><!-- #page-wrapper --> <!--googleoff: all--><div id="cookie-law-info-bar" data-nosnippet="true"><span>This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. <a role='button' tabindex='0' class="cli_settings_button" style="margin:5px 20px 5px 20px;" >Cookie settings</a><a role='button' tabindex='0' data-cli_action="accept" id="cookie_action_close_header" class="medium cli-plugin-button cli-plugin-main-button cookie_action_close_header cli_action_button" style="display:inline-block; margin:5px; ">ACCEPT</a></span></div><div id="cookie-law-info-again" style="display:none;" data-nosnippet="true"><span id="cookie_hdr_showagain">Privacy & Cookies Policy</span></div><div class="cli-modal" data-nosnippet="true" id="cliSettingsPopup" tabindex="-1" role="dialog" aria-labelledby="cliSettingsPopup" aria-hidden="true"> <div class="cli-modal-dialog" role="document"> <div class="cli-modal-content cli-bar-popup"> <button type="button" class="cli-modal-close" id="cliModalClose"> <svg class="" viewBox="0 0 24 24"><path d="M19 6.41l-1.41-1.41-5.59 5.59-5.59-5.59-1.41 1.41 5.59 5.59-5.59 5.59 1.41 1.41 5.59-5.59 5.59 5.59 1.41-1.41-5.59-5.59z"></path><path d="M0 0h24v24h-24z" fill="none"></path></svg> <span class="wt-cli-sr-only">Close</span> </button> <div class="cli-modal-body"> <div class="cli-container-fluid cli-tab-container"> <div class="cli-row"> <div class="cli-col-12 cli-align-items-stretch cli-px-0"> <div class="cli-privacy-overview"> <h4>Privacy Overview</h4> <div class="cli-privacy-content"> <div class="cli-privacy-content-text">This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.</div> </div> <a class="cli-privacy-readmore" aria-label="Show more" tabindex="0" role="button" data-readmore-text="Show more" data-readless-text="Show less"></a> </div> </div> <div class="cli-col-12 cli-align-items-stretch cli-px-0 cli-tab-section-container"> <div class="cli-tab-section"> <div class="cli-tab-header"> <a role="button" tabindex="0" class="cli-nav-link cli-settings-mobile" data-target="necessary" data-toggle="cli-toggle-tab"> Necessary </a> <div class="wt-cli-necessary-checkbox"> <input type="checkbox" class="cli-user-preference-checkbox" id="wt-cli-checkbox-necessary" data-id="checkbox-necessary" checked="checked" /> <label class="form-check-label" for="wt-cli-checkbox-necessary">Necessary</label> </div> <span class="cli-necessary-caption">Always Enabled</span> </div> <div class="cli-tab-content"> <div class="cli-tab-pane cli-fade" data-id="necessary"> <div class="wt-cli-cookie-description"> Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information. </div> </div> </div> </div> <div class="cli-tab-section"> <div class="cli-tab-header"> <a role="button" tabindex="0" class="cli-nav-link cli-settings-mobile" data-target="non-necessary" data-toggle="cli-toggle-tab"> Non-necessary </a> <div class="cli-switch"> <input type="checkbox" id="wt-cli-checkbox-non-necessary" class="cli-user-preference-checkbox" data-id="checkbox-non-necessary" checked='checked' /> <label for="wt-cli-checkbox-non-necessary" class="cli-slider" data-cli-enable="Enabled" data-cli-disable="Disabled"><span class="wt-cli-sr-only">Non-necessary</span></label> </div> </div> <div class="cli-tab-content"> <div class="cli-tab-pane cli-fade" data-id="non-necessary"> <div class="wt-cli-cookie-description"> Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website. </div> </div> </div> </div> </div> </div> </div> </div> <div class="cli-modal-footer"> <div class="wt-cli-element cli-container-fluid cli-tab-container"> <div class="cli-row"> <div class="cli-col-12 cli-align-items-stretch cli-px-0"> <div class="cli-tab-footer wt-cli-privacy-overview-actions"> <a id="wt-cli-privacy-save-btn" role="button" tabindex="0" data-cli-action="accept" class="wt-cli-privacy-btn cli_setting_save_button wt-cli-privacy-accept-btn cli-btn">SAVE & ACCEPT</a> </div> </div> </div> </div> </div> </div> </div> </div> <div class="cli-modal-backdrop cli-fade cli-settings-overlay"></div> <div class="cli-modal-backdrop cli-fade cli-popupbar-overlay"></div> <!--googleon: all--> <div id="fb-root"></div> <script async defer crossorigin="anonymous" src="https://connect.facebook.net/en_US/sdk.js#xfbml=1&version=v12.0&appId=683648729088349&autoLogAppEvents=1"> </script> <!--Start of Tawk.to Script (0.5.5)--> <script type="text/javascript"> var Tawk_API=Tawk_API||{}; var Tawk_LoadStart=new Date(); (function(){ var s1=document.createElement("script"),s0=document.getElementsByTagName("script")[0]; s1.async=true; s1.src='https://embed.tawk.to/5ec04a92967ae56c521a742a/default'; s1.charset='UTF-8'; s1.setAttribute('crossorigin','*'); s0.parentNode.insertBefore(s1,s0); })(); </script> <!--End of Tawk.to Script (0.5.5)--> <script type="text/javascript"> window.WPCOM_sharing_counts = {"https:\/\/machinelearningmastery.in\/2021\/07\/28\/cross-validation-using-oml4py-part-ii\/":1708}; </script> <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> <div id="fb-root"></div> <script>(function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = 'https://connect.facebook.net/en_US/sdk.js#xfbml=1&appId=249643311490&version=v2.3'; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk'));</script> <script> document.body.addEventListener( 'is.post-load', function() { if ( 'undefined' !== typeof FB ) { FB.XFBML.parse(); } } ); </script> <script type="text/javascript"> ( function () { var currentScript = document.currentScript; // Helper function to load an external script. function loadScript( url, cb ) { var script = document.createElement( 'script' ); var prev = currentScript || document.getElementsByTagName( 'script' )[ 0 ]; script.setAttribute( 'async', true ); script.setAttribute( 'src', url ); prev.parentNode.insertBefore( script, prev ); script.addEventListener( 'load', cb ); } function init() { loadScript( 'https://platform.linkedin.com/in.js?async=true', function () { if ( typeof IN !== 'undefined' ) { IN.init(); } } ); } if ( document.readyState === 'loading' ) { document.addEventListener( 'DOMContentLoaded', init ); } else { init(); } document.body.addEventListener( 'is.post-load', function() { if ( typeof IN !== 'undefined' ) { IN.parse(); } } ); } )(); </script> <script id="tumblr-js" type="text/javascript" src="https://assets.tumblr.com/share-button.js"></script> <script type="text/javascript"> ( function () { // Pinterest shared resources var s = document.createElement( 'script' ); s.type = 'text/javascript'; s.async = true; s.setAttribute( 'data-pin-hover', true ); s.src = window.location.protocol + '//assets.pinterest.com/js/pinit.js'; var x = document.getElementsByTagName( 'script' )[ 0 ]; x.parentNode.insertBefore(s, x); // if 'Pin it' button has 'counts' make container wider function init() { var shares = document.querySelectorAll( 'li.share-pinterest' ); for ( var i = 0; i < shares.length; i++ ) { var share = shares[ i ]; if ( share.querySelector( 'a span:visible' ) ) { share.style.width = '80px'; } } } if ( document.readyState !== 'complete' ) { document.addEventListener( 'load', init ); } else { init(); } } )(); </script> <script> (function(r, d, s) { r.loadSkypeWebSdkAsync = r.loadSkypeWebSdkAsync || function(p) { var js, sjs = d.getElementsByTagName(s)[0]; if (d.getElementById(p.id)) { return; } js = d.createElement(s); js.id = p.id; js.src = p.scriptToLoad; js.onload = p.callback sjs.parentNode.insertBefore(js, sjs); }; var p = { scriptToLoad: 'https://swx.cdn.skype.com/shared/v/latest/skypewebsdk.js', id: 'skype_web_sdk' }; r.loadSkypeWebSdkAsync(p); })(window, document, 'script'); </script> <div id="sharing_email" style="display: none;"> <form action="/2021/07/28/cross-validation-using-oml4py-part-ii/" method="post"> <label for="target_email">Send to Email Address</label> <input type="email" name="target_email" id="target_email" value="" /> <label for="source_name">Your Name</label> <input type="text" name="source_name" id="source_name" value="" /> <label for="source_email">Your Email Address</label> <input type="email" name="source_email" id="source_email" value="" /> <input type="text" id="jetpack-source_f_name" name="source_f_name" class="input" value="" size="25" autocomplete="off" title="This field is for validation and should not be changed" /> <img style="float: right; display: none" class="loading" src="https://machinelearningmastery.in/wp-content/plugins/jetpack/modules/sharedaddy/images/loading.gif" alt="loading" width="16" height="16" /> <input type="submit" value="Send Email" class="sharing_send" /> <a rel="nofollow" href="#cancel" class="sharing_cancel" role="button">Cancel</a> <div class="errors errors-1" style="display: none;"> Post was not sent - check your email addresses! </div> <div class="errors errors-2" style="display: none;"> Email check failed, please try again </div> <div class="errors errors-3" style="display: none;"> Sorry, your blog cannot share posts by email. </div> </form> </div> <script data-cfasync="false" type="text/javascript">if (window.addthis_product === undefined) { window.addthis_product = "wpp"; } if (window.wp_product_version === undefined) { window.wp_product_version = "wpp-6.2.6"; } if (window.addthis_share === undefined) { window.addthis_share = {}; } if (window.addthis_config === undefined) { window.addthis_config = {"data_track_clickback":true,"ui_atversion":"300"}; } if (window.addthis_plugin_info === undefined) { window.addthis_plugin_info = {"info_status":"enabled","cms_name":"WordPress","plugin_name":"Share Buttons by AddThis","plugin_version":"6.2.6","plugin_mode":"AddThis","anonymous_profile_id":"wp-2f16336e765908d13c2d341ff0393457","page_info":{"template":"posts","post_type":["post","page","e-landing-page"]},"sharing_enabled_on_post_via_metabox":false}; } (function() { var first_load_interval_id = setInterval(function () { if (typeof window.addthis !== 'undefined') { window.clearInterval(first_load_interval_id); if (typeof window.addthis_layers !== 'undefined' && Object.getOwnPropertyNames(window.addthis_layers).length > 0) { window.addthis.layers(window.addthis_layers); } if (Array.isArray(window.addthis_layers_tools)) { for (i = 0; i < window.addthis_layers_tools.length; i++) { window.addthis.layers(window.addthis_layers_tools[i]); } } } },1000) }()); </script><script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/_inc/build/photon/photon.min.js?ver=20191001' id='jetpack-photon-js'></script> <script src='https://machinelearningmastery.in/wp-includes/js/comment-reply.min.js?ver=5.6.5' id='comment-reply-js'></script> <script id='hoverIntent-js-extra'> var hootData = {"stickySidebar":"disable","contentblockhover":"enable","contentblockhovertext":"disable"}; </script> <script src='https://machinelearningmastery.in/wp-includes/js/hoverIntent.min.js?ver=1.8.1' id='hoverIntent-js'></script> <script src='https://machinelearningmastery.in/wp-content/themes/unos/js/jquery.superfish.min.js?ver=1.7.5' id='jquery-superfish-js'></script> <script src='https://machinelearningmastery.in/wp-content/themes/unos/js/jquery.fitvids.min.js?ver=1.1' id='jquery-fitvids-js'></script> <script src='https://machinelearningmastery.in/wp-content/themes/unos/js/jquery.parallax.min.js?ver=1.4.2' id='jquery-parallax-js'></script> <script id='ap-frontend-js-js-extra'> var ap_form_required_message = ["This field is required","accesspress-anonymous-post"]; var ap_captcha_error_message = ["Sum is not correct.","accesspress-anonymous-post"]; </script> <script src='https://machinelearningmastery.in/wp-content/plugins/accesspress-anonymous-post/js/frontend.js?ver=2.8.1' id='ap-frontend-js-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/hootkit/assets/jquery.lightSlider.min.js?ver=1.1.2' id='jquery-lightSlider-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/hootkit/assets/widgets.min.js?ver=2.0.8' id='hootkit-widgets-js'></script> <script id='hootkit-miscmods-js-extra'> var hootkitMiscmodsData = {"ajaxurl":"https:\/\/machinelearningmastery.in\/wp-admin\/admin-ajax.php"}; </script> <script src='https://machinelearningmastery.in/wp-content/plugins/hootkit/assets/miscmods.min.js?ver=2.0.8' id='hootkit-miscmods-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/page-links-to/dist/new-tab.js?ver=3.3.5' id='page-links-to-js'></script> <script src='https://s7.addthis.com/js/300/addthis_widget.js?ver=5.6.5#pubid=ra-5e0c443d44eeeb15' id='addthis_widget-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/vendor/automattic/jetpack-lazy-images/src/js/intersectionobserver-polyfill.min.js?ver=1.1.2' id='jetpack-lazy-images-polyfill-intersectionobserver-js'></script> <script id='jetpack-lazy-images-js-extra'> var jetpackLazyImagesL10n = {"loading_warning":"Images are still loading. Please cancel your print and try again."}; </script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/vendor/automattic/jetpack-lazy-images/src/js/lazy-images.min.js?ver=1.1.2' id='jetpack-lazy-images-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/_inc/build/postmessage.min.js?ver=9.8.1' id='postmessage-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/_inc/build/jquery.jetpack-resize.min.js?ver=9.8.1' id='jetpack_resize-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/_inc/build/likes/queuehandler.min.js?ver=9.8.1' id='jetpack_likes_queuehandler-js'></script> <script src='https://machinelearningmastery.in/wp-content/themes/unos/js/hoot.theme.min.js?ver=2.9.11' id='hoot-theme-js'></script> <script src='https://machinelearningmastery.in/wp-content/plugins/youtube-embed-plus/scripts/fitvids.min.js?ver=14.0' id='__ytprefsfitvids__-js'></script> <script src='https://machinelearningmastery.in/wp-includes/js/wp-embed.min.js?ver=5.6.5' id='wp-embed-js'></script> <script id='jetpack_related-posts-js-extra'> var related_posts_js_options = {"post_heading":"h4"}; </script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/_inc/build/related-posts/related-posts.min.js?ver=20210604' id='jetpack_related-posts-js'></script> <script id='sharing-js-js-extra'> var sharing_js_options = {"lang":"en","counts":"1","is_stats_active":"1"}; </script> <script src='https://machinelearningmastery.in/wp-content/plugins/jetpack/_inc/build/sharedaddy/sharing.min.js?ver=9.8.1' id='sharing-js-js'></script> <script id='sharing-js-js-after'> var windowOpen; ( function () { function matches( el, sel ) { return !! ( el.matches && el.matches( sel ) || el.msMatchesSelector && el.msMatchesSelector( sel ) ); } document.body.addEventListener( 'click', function ( event ) { if ( ! event.target ) { return; } var el; if ( matches( event.target, 'a.share-facebook' ) ) { el = event.target; } else if ( event.target.parentNode && matches( event.target.parentNode, 'a.share-facebook' ) ) { el = event.target.parentNode; } if ( el ) { event.preventDefault(); // If there's another sharing window open, close it. if ( typeof windowOpen !== 'undefined' ) { windowOpen.close(); } windowOpen = window.open( el.getAttribute( 'href' ), 'wpcomfacebook', 'menubar=1,resizable=1,width=600,height=400' ); return false; } } ); } )(); var windowOpen; ( function () { function matches( el, sel ) { return !! ( el.matches && el.matches( sel ) || el.msMatchesSelector && el.msMatchesSelector( sel ) ); } document.body.addEventListener( 'click', function ( event ) { if ( ! event.target ) { return; } var el; if ( matches( event.target, 'a.share-telegram' ) ) { el = event.target; } else if ( event.target.parentNode && matches( event.target.parentNode, 'a.share-telegram' ) ) { el = event.target.parentNode; } if ( el ) { event.preventDefault(); // If there's another sharing window open, close it. if ( typeof windowOpen !== 'undefined' ) { windowOpen.close(); } windowOpen = window.open( el.getAttribute( 'href' ), 'wpcomtelegram', 'menubar=1,resizable=1,width=450,height=450' ); return false; } } ); } )(); </script> <iframe src='https://widgets.wp.com/likes/master.html?ver=202142#ver=202142' scrolling='no' id='likes-master' name='likes-master' style='display:none;'></iframe> <div id='likes-other-gravatars'><div class="likes-text"><span>%d</span> bloggers like this:</div><ul class="wpl-avatars sd-like-gravatars"></ul></div> <script>!function(){window.advanced_ads_ready_queue=window.advanced_ads_ready_queue||[],advanced_ads_ready_queue.push=window.advanced_ads_ready;for(var d=0,a=advanced_ads_ready_queue.length;d<a;d++)advanced_ads_ready(advanced_ads_ready_queue[d])}();</script><script src='https://stats.wp.com/e-202142.js' defer></script> <script> _stq = window._stq || []; _stq.push([ 'view', {v:'ext',j:'1:9.8.1',blog:'170785677',post:'1708',tz:'-5.5',srv:'machinelearningmastery.in'} ]); _stq.push([ 'clickTrackerInit', '170785677', '1708' ]); </script> </body> </html> <!-- Page generated by LiteSpeed Cache 4.4.3 on 2021-10-20 08:51:30 -->