RUN CASES ON AZURE

Summary

Run Cases on Azure submits GeoClaw cases that you created in your specified working directory using the Create GeoClaw Cases tool for asynchronous execution on your Microsoft Azure^® Batch account.

To learn more about the Liquids HCA Tool in general, please see Liquids HCA Tool Frequently Asked Questions.

To learn more about the structure of the Liquids HCA Tool project geodatabase, please see Liquids HCA Tool Data Dictionary.

Usage

The Liquids HCA Tool uses a customized version of an open source Computational Fluid Dynamics (CFD) modeling package called GeoClaw to perform overland flow plume simulations (Berger, M.J., George, D.L., LeVeque, R.J. and Mandli, K.M. (2011) The GeoClaw software for depth-averaged flows with adaptive refinement. Advances in Water Resources, 34, pp. 1195 – 1206). GeoClaw uses high-resolution, Godunov-type, shock-capturing finite volume methods and Riemann solvers to solve the shallow water equations, a simplified, depth-integrated version of the general Navier-Stokes equations for incompressible, viscous fluid flow. Of particular importance to modeling pipeline releases and storage tank ruptures, dry states in the model space are handled automatically by GeoClaw. GeoClaw incorporates Adaptive Mesh Refinement (AMR) to facilitate efficient and stable simulation of the highly transient flows typical of pipeline releases and storage tank ruptures. The G2-IS version of GeoClaw replaces the Manning equation in GeoClaw for friction factor calculation with the more generic Darcy-Weisbach equation, which allows for the incorporation of fluid density and viscosity terms, facilitating the modeling of working fluids other than water, such as hydrocarbons.

Run Cases on Azure submits GeoClaw cases for execution on your Azure Batch account; it also instantiates the Azure resources needed to execute your GeoClaw cases if they do not already exist on your Batch account. These resources include:

On your Azure Batch account –
- A batch job named landspill-azure-job to which your GeoClaw cases are added as individual tasks.
- A computing pool named landspill-azure-pool; the tasks (GeoClaw cases) in landspill-azure-job are executed on the nodes in this pool.
On your Azure Batch account’s associated Storage account –
- An Azure BLOB storage container named landspill-azure-container that stores the task output (in individual case folders) from landspill-azure-job.
- An Azure NoSQL table named TABLElandspillazure that tracks landspill-azure-job task status.

Run Cases on Azure requires you to specify the following parameters to submit your GeoClaw cases and to instantiate the necessary Azure resources as needed:

Working Directory – This must be the same working directory that you used in running the Create GeoClaw Cases tool.

Input Release Point Features – This should be the same feature layer that you used in the Create GeoClaw Cases tool. However, if you wish to submit only a subset of the cases you created, you may do so by applying either a definition query or a feature selection to your release point feature layer.
Release Point Identifier Field – A field that uniquely identifies your input release point features. This should be the same as the field name specified in the Create GeoClaw Cases The default value is the POINT_ID field.
Maximum Number of Computing Nodes – This parameter allows you to specify the desired number of computing nodes in the landspill-azure-pool pool in your Azure batch account. The default value is two (2). You should make sure your number of computing nodes is consistent with your Azure Batch account’s quota for virtual CPUs (vCPUs) and the type of computing node you specify. If you specify a number of computing nodes that exceeds your Batch account’s vCPU quota, the number of nodes will be automatically adjusted downward by Azure to a number consistent with your quota.

The tool only creates the landspill-azure-pool pool if it does not already exist. If the landspill-azure-pool pool already exists, the tool will not modify it (and the value in this parameter will have no effect). If you wish to resize the landspill-azure-pool pool, you should first delete it, either in Azure or by using the Delete Azure Resources tool.

Computing Node Type – This parameter is used to specify the virtual machine node type for use in running your GeoClaw simulations on the landspill-azure-pool pool in your Azure batch account. You should make sure that your Azure Batch account quota is configured to accept your selected node type. Unless otherwise specified, the tool defaults this parameter to the value ‘STANDARD_H8,’ a high-performance 8-core virtual machine recommended by G2-IS for use in GeoClaw processing. To learn more about this virtual machine, please see the Azure help topic, H-series.

The tool only creates the landspill-azure-pool pool if it does not already exist. If the landspill-azure-pool pool already exists, the tool will not modify it (and your selected node type will have no effect). If you wish to reinstantiate the landspill-azure-pool pool with a different node type, you should first delete it, either in Azure, or by using the Delete Azure Resources tool.

Azure Credential Method – You can either use the encrypted, passcode-protected Azure credential file that you created using the Create Encrypted Azure Credential File tool or you can manually input your Azure Batch and associated Storage account credentials. When you select “Encrypted file” for this parameter the following required dependent parameters are exposed:
- Encrypted Credential File – This parameter is the path to your encrypted, passcode-protected credential file. By default, the tool looks for a credential file named azure_cred.bin in the scratch folder for the active ArcGIS Pro project.
- Encrypted Credential File Passcode – This parameter requires you to specify the passcode that you supplied when creating your encrypted Azure credential file. The encrypted Azure credential file can only be unencrypted for use when you supply this passcode. You should apply the same security measures for safeguarding this passcode that you apply to your Windows password.

The following required dependent parameters are exposed when you select “Manual input” for the Azure Credential Method parameter:

- Azure Batch Account Name – Your Azure Batch account name.
- Azure Batch Account Key – The primary or secondary access key for your Azure Batch account.
- Azure Batch Account URL – The URL for your Azure Batch account.
- Azure Storage Account Name – The name of the Azure Storage account associated with your Azure Batch account.
- Azure Storage Account Key – Key1 or Key2 for the Azure Storage account associated with your Azure Batch account.
Azure Pool Docker Image – This parameter is used to specify the Docker™ machine image that will be used by Azure to instantiate your Azure Batch account pool nodes for GeoClaw processing. By default, this parameter is set to the g2integratedsolutions/landspill:g2bionic1_2 machine image hosted on Docker Hub. There generally should be no need to alter this parameter.
Do not resubmit a case if it already exists on Azure – This checkbox parameter allows you to specify whether to overwrite a GeoClaw case on Azure for which output already exists. By default, this checkbox parameter is checked, meaning that the tool will not resubmit and overwrite an existing case on Azure.

Both your Azure Batch account quotas and the number of cases submitted affect the time it takes for the cases submitted to Azure by the Run Cases on Azure tool to fully execute. You can either use the Monitor Azure Resources tool or the Azure portal interface to monitor progress.

In a typical Liquids HCA Tool workflow, Run Cases on Azure is run after Create GeoClaw Cases and before Download Cases from Azure.

For visual reference on Liquids HCA Tool execution order, see Liquids HCA Tool Process Flow Diagrams.

Syntax

RunCasesOnAzure (working_dir, rupture_point, case_name_field, max_nodes, vm_type, cred_type, {cred_file}, {passcode}, {azure_batch_name}, {azure_batch_key}, {azure_batch_URL}, {azure_storage_name}, {azure_storage_key}, azure_pool_docker_image, ignore_azure_exist)

Parameter	Explanation	Data Type
working_dir	Dialog Reference Specify the directory that stores the GeoClaw cases associated with your release points. There is no Python reference for this parameter.	Workspace
rupture_point	Dialog Reference Specify the release point features associated with your GeoClaw cases. There is no Python reference for this parameter.	Feature Layer
case_name_field	Dialog Reference Select the field storing your release point unique identifier values. There is no Python reference for this parameter.	Field
max_nodes	Dialog Reference Specify the maximum number of computing Azure Batch account computing nodes. There is no Python reference for this parameter.	Long
vm_type	Dialog Reference Select the Azure Batch account computing node type. There is no Python reference for this parameter.	String
cred_type	Dialog Reference Select the Azure authentication method. There is no Python reference for this parameter.	String
cred_file (Optional)	Dialog Reference Select your encrypted, passcode-protected Azure credential file. There is no Python reference for this parameter.	File
passcode (Optional)	Dialog Reference Specify the passcode for your encrypted, passcode-protected Azure credential file. There is no Python reference for this parameter.	String Hidden
azure_batch_name (Optional)	Dialog Reference Specify the name of your Azure Batch account. There is no Python reference for this parameter.	String Hidden
azure_batch_key (Optional)	Dialog Reference Specify your Azure Batch account key. There is no Python reference for this parameter.	String Hidden
azure_batch_URL (Optional)	Dialog Reference Specify your Azure Batch account URL. There is no Python reference for this parameter.	String Hidden
azure_storage_name (Optional)	Dialog Reference Specify your Azure Storage account name. There is no Python reference for this parameter.	String Hidden
azure_storage_key (Optional)	Dialog Reference Specify your Azure Storage account key. There is no Python reference for this parameter.	String Hidden
azure_pool_docker_image	Dialog Reference Specify your Azure pool Docker machine image. There is no Python reference for this parameter.	String
ignore_azure_exist	Dialog Reference Check this box if you do not wish to overwrite existing GeoClaw case results on Azure. There is no Python reference for this parameter.	Boolean

Code sample

The following script demonstrates how to use Run Cases on Azure with an encrypted, passcode-protected file:

import arcpy

arcpy.ImportToolbox(r”C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\liquidshca\esri\toolboxes\LiquidsHCA.pyt”)

in_dir = r”C:\data”

arcpy.env.workspace = r”C:\data\test.gdb”

release_points = “OSPointM”

point_id_field = “POINT_ID”

no_of_notes = 2

node_type = “STANDARD_H8”

cred_type = “Encrypted file”

file_path = r”C:\data\azure_cred.bin”

passcode = “XXXXXXX”

batch_name = None

batch_key = None

batch_url = None

storage_name = None

storage_key = None

azure_pool_docker_img = “g2integratedsolutions/landspill:g2bionic1_1”

no_resub_case = True

arcpy.liquidshca.RunCasesOnAzure(in_dir, release_points, point_id_field, no_of_notes, node_type, cred_type, file_path, passcode, batch_name, batch_key, batch_url, storage_name, storage_key, azure_pool_docker_img, no_resub_case)

Environments

Scratch Workspace

Licensing information

This tool requires a valid Liquids HCA Tool user license or subscription. Please see the Request License and Register License tool help topics for details on obtaining and registering a Liquids HCA Tool software license. The version of GeoClaw that is customized to use the Darcy-Weisbach friction factor is available here for use under the terms of the Berkeley 3-Clause (BSD-3-Clause) open source license. The Create GeoClaw Cases tool is a G2-IS customized version of a tool present in the geoclaw-azure-launcher tools available here for use under the terms of the Berkeley 3-Clause (BSD-3-Clause) open source license.

Credits

geoclaw-azure-launcher:

GeoClaw:

Use limitations

There are no access and use limitations for this item.