Understand Analytic Ingestion

About Analytic Ingestion Services

Application Analytics provides specific REST API services to upload or ingest your analytic logic, template definitions, and deployments simultaneously.

You can use either the Analytic UI or any REST client such as Postman to perform the ingestion. However, before you can prepare the .zip file for upload, it is important to understand the configuration, schema definitions, and structure for the expected file for upload or ingestion.

Predix Essentials Analytics has three API endpoints, facilitating three different methods of analytic data ingestion.

Table 1. Predix Essentials Analytic Ingestion Service Endpoints
Service Endpoint	Description
/analytics/ingestions	This facilitates ingesting all analytic-related data to run deployments. This method requires you to ingest the analytic template, template configurations including supporting files, input, output, and output event definitions. This also requires you to ingest deployment configurations required to run the deployment in the specified analytic runtime. Note: Currently this method is only available through the service endpoint. Ingesting through the Analytic UI does not use this endpoint.
/analyticEntries/ingestions	This facilitates ingesting the following files: Analytic template configurations Supporting mapping files for deployment (optional) This requires you to ingest the analytic template, template configurations including supporting files, input, output, and output event definitions. This method is currently supported both through the UI and service endpoint.
`/deployments/ingestions`	This facilitates ingesting deployment configurations only. This assumes that you have already pre-ingested at least one analytic template for which you can add and configure deployments. This requires you to ingest deployment configurations required to run the deployment in the specified analytic runtime. Note: Currently this method is only available through the service endpoint. Ingesting through the Analytic UI does not use this endpoint.

Expected Folder Structure

The following diagram represents the expected file and folder structure for an analytic ingestion .zip file. All folders and files indicated below are required for ingestion through the /analytics/ingestions endpoint. The diagram indicates the required folders and files for ingestion through the /analyticEntries/ingestions endpoints in blue. It indicates required folders and files for ingestion through the /deployments/ingestions endpoint in teal.

Figure: Ingestion File Structure Diagram

Schema Definition for Analytic Ingestion Files

Each analytic .json file intended to be uploaded through the analytic ingestion services must conform to the respective schemas represented in this document.

AnalyticDefinition.json

This file defines the ingestion metadata contract for AnalyticEntry. This file contains the metadata information for the analytic template such as name, URI, author, and contract version. It also has the analytic template configuration and definitions including inputs, outputs, constants, and supporting files with relative paths to the analytic definition file. Application Analytics does not validate the file name uniqueness and expects the user to keep track of file names. The analytic template metadata and configurations file, for example, analyticdefinition.json, must be placed inside the analyticEntries folder.

The following code snippet indicates the JSON structure with required (mandatory) field configurations. The file must conform to the schema indicated in the code snippet.

{
  "@type": "analyticEntrySsRmMetaData", /* analyticEntrySsRmMetaData is for RM, analyticEntrySsMetaData for MEH , analyticEntrySparkMetaData for Spark*/ //Mandatory
  "contractVersion":"1.0.0", // Mandatory
  "name": "Analytic1", //Mandatory
   "uri": "/analytics/16a47853-0804-4972-ac33-b16d33336352", // Not always required. In case of Smart Signal, this is provided. CAF preserves it. Mandatory for SmartSignal
  "author": "GE Digital - Smart Signal Analytics Team", //Mandatory
  "description": "This is UC_Temperature_Asset's description:",
  "analyticRuntimeType": "SmartSignal", /* will be Smart Signal */ //Mandatory, Same for bith MEH and RM
  "primaryCategory": "Supervised Learning", //Mandator
  "secondaryCategory": "Supervised Learning", /* OPTIONAL */
  "analyticVersion": "1.0.0",
  "hasAnalyticArtifact": false, //Mandatatory
    "analyticArtifact": {   //Mandatory
        "language": "SmartSignal",
        "languageVersion": "v1"
    },
    "executeParams": { // This is specific to Smart Signal and execute Params are not required. CAF uses this as a default for Smart Signal deployments.
        "repeatData": {
            "timeValue": 1,
            "timeUnit": "MINUTES"
        }
    },
  "isReadyOnly": true
  // START Mandatory
  "analyticDefine": {
    "supportingFiles": [
      {
        "filename": "../supportingFiles/blueprint.json",
        "description": "This is UC_Temperature_Asset's description:"
      },
      {
        "filename": "../supportingFiles/modeblueprints.json",
        "description": "Mode Blueprints"
      },
      {
        "filename": "../supportingFiles/modelblueprints.json",
        "description": "Model Blueprints"
      },
      {
        "filename": "../supportingFiles/assetblueprinttags.json",
        "description": "Asset Blueprint Tags"
      },
      {
        "filename": "../supportingFiles/modelblueprinttags.json",
        "description": "Model Blueprint Tags"
      },
      {
        "filename": "../supportingFiles/assetblueprinttagrules.json",
        "description": "Asset Blueprint Tag Rules"
      },
      {
        "filename": "../supportingFiles/modelblueprinttagrules.json",
        "description": "Model Blueprint Tag Rules"
      },
      {
        "filename": "../supportingFiles/ruletagassociations.json",
        "description": "Rule Tag Associations"
      },
      {
        "filename": "../supportingFiles/diagnostics.json",
        "description": "Diagnostic Rules"
      }
    ],
    "inputs": [
      {
        "name": "temperature",
        "unit": "",
        "description": "temperature",
        "dataType": "Double"
      },
      {
        "name": "pressure",
        "unit": "",
        "description": "pressure",
        "dataType": "Double"
      }
    ],
    "constants": [
      {
        "value": 0.92,
        "uri": "",
        "name": "threshold",
        "unit": "",
        "description": "threshold",
        "dataType": "Double"
      }
    ],
    "outputs": [
      {
        "name": "power",
        "unit": "",
        "description": "",
        "dataType": "Double"
      },
      {
        "name": "heatrate",
        "unit": "",
        "description": "",
        "dataType": "Double"
      },
      {
        "name": "availability",
        "unit": "",
        "description": "",
        "dataType": "Double"
      }
    ]
    // END Mandatory
    "outputEvents" :[
      "name": "<your alert template name. Must be present in the tenant. Validated. >",
    ]
  }
}

deployments.json

This file defines the ingestion metadata contract for all deployments. Each deployment .json file must begin with the prefix deployment_. This file contains the following information required to configure analytic deployments:

Link to template artifact
Name for this deployment
Author of this deployment
Asset information - which asset this analytic is applied to
Link to IOMapping artifact files.

The following code snippet indicates the .json structure with required (mandatory) field configurations. The file must conform to the schema indicated in the code snippet.

{
    @type: "deploymentSsRmMetadata", // Mandatory
    "contractVersion": "1.0.0", //Mandatory
    "name":"<your deployment name>", //Mandatory
    "templateType": "Analytic", /* Analytic or Orchestration */ //Mandatory
    "templateName": "<your template name from the template.json ingested" //Mandatory
   "templateVersion": "<version>",
    "isReadOnly":true, //Mandatory - If set to false, the deployment configuratons can be modified in the Analytic UI.
    "keyValueStore" :{ 
        "runtimeAnalyticUri":"<your url CAF will call to get custom data for deployment>", //Mandatory for RM Smart Signal Ex: /analyticIntance/13a47853-0804-4972-ac33-b16d33336351)
        targetCardType:"SmartSignal", //Mandatory
        "<key1>":"<value>",
        ...
    },
   "entityFilterFileName": "../entityFilter.json",   //Mandatory, this is the relative path of the entityFilter.json
     
  "deploymentSteps": [
    {
        "analyticName": "<name>",    //Mandatory for Template Type "Orchestration"
       "analyticVersion": "<analyticVersion>",
        "ioMappingFiles": [
            "../ioMappings/IOMappingMetadata1.json",
            "../ioMappings/IOMappingMetadata2.json" // Multiple only if there is iterations.
        ]
      }
]
  "deployParams": { /* Scheduling and data parameter information can be given here, This is not required in case of RM analytics */
    "analyticArtifactVersion": null,
    "repeatData": null,
    "startTime": "1492723560",
    "endTime": "1493069160",
    "samplingInterval": null,
    "sampleDuration": null,
    "startOffset": null,
    "deployType": "DeployOnDemand",
    "doExecuteJob":false,
    "doDeployPerAsset": true
  },
   
  "doDeploy": true
}

ioMapping.json

Contains the mapping information for inputs and outputs in the template to asset tags.

The following code snippet indicates the .json structure with required (mandatory) field configurations. The file must conform to the schema indicated in the code snippet.

{
  "@type": "ioMappingSsRmMetaData", // Mandatory
  "contractVersion": "1.0.0", //Mandatory
  "name": "ioMappingSsRm1", // Mandatory
  "description": "description",
   "inputsToDataSource": [  //Mandatory Start here
        {
            "portName": "EXHAUST_STEAM_TEMP_1",
            "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "EXHAUST_STEAM_TEMP_1"
            },
            "assetSourceKeyToDataSources": {}
        },
        {
            "portName": "GROSS_POWER",
            "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "GROSS_POWER"
            },
            "assetSourceKeyToDataSources": {}
        },
        {
            "portName": "INLET_STEAM_TEMP_1",
            "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "INLET_STEAM_TEMP_1"
            },
            "assetSourceKeyToDataSources": {}
        },
        {
            "portName": "MODE_TAG_1",
            "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "MODE_TAG_1"
            },
            "assetSourceKeyToDataSources": {}
        },
        {
            "portName": "INLET_STEAM_PRESS_1",
            "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "INLET_STEAM_PRESS_1"
            },
            "assetSourceKeyToDataSources": {}
        },
        {
            "portName": "EXHAUST_STEAM_PRESS_1",
            "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "EXHAUST_STEAM_PRESS_1"
            },
            "assetSourceKeyToDataSources": {}
        }
    ],
    "constantsToDataSource": [
        {
            "portName": "TURBINE_EXHAUST_STEAM_QUALITY",
            "commonDataSource": {
                "@type": "ioDataSourceValueCommon",
                "dataType": "Integer",
                "value": 0.92
            },
            "assetSourceKeyToDataSources": {}
        },
        {
            "portName": "MODE_TAG_1_THRESHOLD",
            "commonDataSource": {
                "@type": "ioDataSourceValueCommon",
                "dataType": "Double",
                "value": 30.0
            }
        }
    ],
    "outputsToDataSource": [
        {
            "portName": "MODE_CV",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "MODE_CV"             
            }
        },
        {
            "portName": "TURBINE_PRESS_RATIO_CV",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "TURBINE_PRESS_RATIO_CV"
            }
        },
        {
            "portName": "TURBINE_EXHAUST_STEAM_QUALITY_CV",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "TURBINE_EXHAUST_STEAM_QUALITY_CV"
            }
        },
        {
            "portName": "TURBINE_EFFICIENCY_CV",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "TURBINE_EFFICIENCY_CV"
            }
        },
        {
            "portName": "TURBINE_TEMP_RATIO_CV",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "TURBINE_TEMP_RATIO_CV"
            }
        },
        {
            "portName": "CVMEHtag",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "CVMEHtag"
            }
        },
        {
            "portName": "CVMEhTagF",
             "commonDataSource": {
                "@type": "ioDataSourceTagCommon",
                "entityType": "Asset",
                "tagName": "CVMEhTagF"
            }
        }
    ] //Mandatory Ends here
}

entityFilter.json

This entity filter is created and is referenced in the deployment as the asset selection for the deployment.

The following code snippet indicates the .json structure with required (mandatory) field configurations. The file must conform to the schema indicated in the code snippet.

{
    "contractVersion": "1.0.0",
    "entityFilter": {
        "name": "XE7FA_297479_3504_Ingestion",
        "entityFilterConditions": [
            {
                "entityFilterFieldName": "Asset Name",
                "operator": "=",
                "value": "XE7FA_297479_3504"
            }
        ]
    }
}