Delta Watch: Import & Export module

Need to send processed Delta Watch data to a third party?

Importing geotechnical sensor data?

The Import & Export module can take care of that.

This guide explains the ease of use and flexibility that we put in your hands.

Log in or sign up for myTopcon to view this page:

Delta Watch: Import & Export module

Introduction

Delta Watch has got a highly user configurable Import & Export module. With this module, it is possible to use a predefined export format or to define your own. The last is particularly useful when there is a need to provide third parties with a continuous processed data flow of all sorts.

The module can be found in the main access bar on the left by navigating to Interfaces > Import & Export.

In this module, you can add an unlimited number of import and export configurations, called tasks.

Each job can be configured to execute at specific time intervals and will run independent of other processing tasks. The custom format configuration is very flexible and allows you to configure things like:

  • Sensor type
  • Filename
  • Date/time format
  • Headers
  • Column separator
  • Data values
  • Units
  • Decimal places

The import capabilities of this module are used for items like:

  • ENV or THP sensor import (*.ENV or *.THP)
  • Delta Link control unit information (*.CUI)
  • Total station information file statistics (*.STNI)
  • Geotechnical sensors (Tilt, water level, inclinometer, etc.)

The export options are endless and can be customized to fit the project specific needs.

Additionally, alarms can be configured to trigger events, such as when no data has been sent or received for a certain amount of time. The Import & Export module also provides a status overview showing, for example, age of last action. The image below shows the status overview in the module. Similar, but less detailed, info is available in Delta Live! on the dashboard. This will only show the last time data was imported or exported.

Create custom export formats

The Import & Export module has a few predefined export formats. In Delta Watch versions older than v2.15, the export was done from within the Automated networks module. That method is now obsolete, and the new module should be used. The table below shows details on the geodetic export formats, their new name, and what they were called in the older versions.

Geodetic Data Type Export Formats
Export Type Replaces Export Option (pre-2.15 versions)
GeodeticPointExport  
Kronos_3Ddeformation Export to information system with export format set to Kronos*
Kronos_Levelling (leveling section) Export to information system with export format set to Kronos*
Iris(Group; Name) Export to information system with export format set to Iris(Group; Name)*
Iris(Name; Group) Export to information system with export format set to Iris(Name; Group)*
ProcessedPointExport Option Processed Point Export

 

* The export type was normally set at Delta Watch > System > Automatic_processing > General >  Export_type.

Upgraded from a Delta Watch pre v3.x?

Make sure you have disabled ALL network exports before upgrading. Alternatively, log in to Delta Watch as Administrator, and navigate to Administration > System configuration > Support. Here, use the function to deactivate all exports from geodetic networks.

Most data export was done from Automated networks, but if you were exporting from other modules, then you will find each format back in the export module once you select the Sensor type.

 

Configure the format

A format is a sort of template that tells what data should go in which columns, and, once defined, can be used by multiple jobs. Besides the predefined formats, users can now define their own format. Follow the simple steps below for how to do this.

  1. Navigate to Interfaces > Import & Export module
  2. Click the Custom format button in the top toolbar. A new dialog box, like the figure below, opens, showing all current available formats.
  1. Click either Add Add or Duplicate Copy. To see an output example of the existing formats, select the item of interest, and the overview will update with the specific details.
  2. A dialog box opens where a new Custom format can be configured. The dialog box will look something like the image on the right. All fields need to be configured. The table gives some details here.
  3. Click OK to save the custom format configuration. This can now be used for automated or manual exports.
Custom Export Format - Configurable Fields
Setting Value
Name A user-chosen name for this format definition
Filename extension Currently only .CSV export possible
Sensor type

Selecting the correct type will ensure only applicable data fields are shown and that the definition only shows when creating an export for this sensor type. The following options are currently available:

  1. Geodetic sensor
  2. Meteorological
  3. Structural monitoring
  4. Convergence section
Import/Export

Defines the format type. Currently only Export can be selected.

File name The filename format, which can be customized. See below for more details on options available.
Column separator

Specify the character to use. Enter [TAB] for the tab character.

Decimal separator

Select either dot or comma to be used.

Timestamp format The timestamp format to be used, which can be customized. See below for more details.
Header configuration

If enabled, then a header will be written as the first line of each data file. This can be an automatic header based on the chosen data columns or a custom header.

Column configuration
  1. Add/remove columns as required. The action is always performed on the last column. Inserting or selective delete is not possible.
  2. For each column where applicable, specify:
    1. Column type
    2. Unit
    3. Decimal places
    4. Text (only when fixed text is selected)

 

File name and timestamp formats

Select an existing formatting using the drop-down control or add/edit a format. To add or edit, click on Add format, and a dialog box will open where existing and new formats can be managed. When using Edit Editor Add [Add], a dialog box shows with all options that can be used, as per the image below. When done, be sure to select the format using the drop-down control as in the above image.

Configure automatic export

The export jobs are replacing all automated exports from pre Delta Watch v2.15 versions. To create an export job, follow the below steps:

  1. Add a new job definition by clicking on the Add Add button.
  1. A dialog box will open where the new task can be defined. The table below, Export Task - Configurable Fields, clarifies the available fields, and the below image shows two typical configuration examples.
  2. Click OK to save the new configuration, and then click Apply to write to the database.
Export Task Definition - Configurable Fields
Setting Value
Name Memorable name for this export. When naming tasks, keep in mind that you can only export from one project but can export multiple Point groups at the same time!
Import/Export Set to Export.
Sensor type

The following options are currently available:

  1. Geodetic sensor
  2. Meteorological
  3. Structural monitoring
  4. Convergence section
  5. Rail track
  6. Tilt meter
Export folder

Specify the destination folder. Ensure a unique export folder per task so that data is not accidentally mixed or overwritten.

Time interval
Start time
Export after processing

Together these fields specify when data export should happen. Enable Extended settings to see all options. The Start time has two functions:

  1. Set a date in the future so the job will not start until reached.
  2. Specifying the time to control base time of the export. For example, setting the time to 00:10:00 with a 60 minutes Time interval, the job will export at 00:10:00; 01:10:00, 02:10:00, 03:10:00…….23:10:00.

Export after processing is available for some export types and, when enabled, the Time interval field is ignored, and export is triggered by the respective module after processing new data.

Used export time zone By default, the timestamp of the exported data is in UTC. Change this field if the date/time stamp should reflect a different timezone.
Export type A list with export formats. Any custom export formats for the selected sensor type will also be visible here.
Select project

Select from which project the data will be exported

Select groups
Select sensors

Depending on which sensor type is exported, either a sensor-group or a sensor-field might need to be selected. Multiple entries can be selected, but if selecting point groups, all the data will be written to the same file unless the filename definition has the [Point Group] field enabled. Otherwise this can give unwanted results, like duplicate entries.

Monitor data transfer

Complete this section to get a message on certain actions. The email server needs to be configured, and the project must have one or more Contacts. The following triggers are available:

  1. Data transfer
  2. Incorrect Export
  3. Confirmation (that data was exported)

Configure automatic import

Import configurations are used to import external data. This could be the special data files that Delta Link generates, such as:

  • Basic weather station data (*.ENV or *.THP)
  • Control unit information (*.CUI)
  • Station information (*.STNI)

Or data files from other geotechnical sensors, such as below but not limited to:

  • Tilt meters
  • Water level sensors
  • Inclinometers

This article is limited to explaining the general setup of an Import task. For more details on each of the above items, there are specific articles which will explain the complete setup for each item.

  1. Add a new job definition by clicking on the Add   button in the Import & Export module. A dialog box shows similar as for exporting.
  2. Follow the order of the fields in the below table to complete the configuration.
  3. When completed, close the dialog box, and click Apply to save the changes.

 

For most sensors only one import definition per project and/or import folder needs to be created. This will automatically process all compatible files in the specified import folder.
Custom Import Task - Configurable Fields
Setting Value
Name A user-chosen name for this format definition
Import/Export Set to Import
Sensor type Set to correct sensor type
Base directory

Directory where the data files can be found

Time interval

How often Delta Watch should check for new data. If this is for CUI data and alarms are configured, then a relatively high interval of 5 or 10 minutes might need to be set.

Import type

Depending on the sensor type, there might be more than one format available. Make sure you select the correct format. More info could be found in sensor-specific articles elsewhere.

Current available imports are:

  1. AMTS Import, imports *.CUI and *.STNI
  2. Meteorological, imports *.ENV or *.THP
  3. Extensometer, imports custom GeoScope or .CSV
  4. Waterlevel, imports custom GeoScope or .CSV
  5. Tilt meter, imports Senceive, custom GeoScope, or VMT .CSV format
Select project The destination project for which this import is
Message to

If alarms are configured for any sensor(s) that are imported by this task, then select here who should receive the emails. One or more project contacts would need to be defined to be able to select them.

Comment

Allows the user to enter some information here

 

Extended settings (optional)

These settings allow a little more control and configuration of the data, especially if the data is old data or created with local timestamps instead of the standard expected UTC time.

Setting Value
Active

Allows activating/deactivating this import definition

Start time Specify the time for the start of the data. Normally this is in the future but could be a time in the past matching the date/time from which point onwards the data needs to be processed. Any data from before this time will not be processed.
Data time zone If the timestamp in the data files is NOT in the UTC time zone, then specify here the correct time zone so that Delta Watch can convert the timestamp information correctly. By default, Delta Watch assumes the data is in UTC time zone.

 

Monitor data transfer (optional)

It is possible to configure email messages triggered by the data import definition. For example, an email could be sent when data has not arrived for a certain amount of time. This can be configured here. Click on [add] to Add one of the project recipients and define the amount of time after which the message should go out. If the list of recipients is empty, then configure these first in the Project > Contacts section.

 

Finished

Once all is set up and saved to the database, Delta Watch might, depending on the start time and interval settings, already start with the import. The Import status can always be seen in this module in the columns Latest datatransfer and Total datatransfer. An example of this can be seen in the figure below.

Using the Manual export function

From this Import & Export module, you can also perform manual exports of data. It is possible to export data from a specific time in a specific (user-defined) format to a user-defined folder.

  1. In the Import & Export module, click on the Manual export button in the main toolbar. See image.
  1. A new dialog box will open up. Make the configuration for the following fields:
    • Select project
    • Sensor type
    • Export type
    • Export folder
    • Used timezone
  2. Depending on the selected Sensor type and the current definitions in your database, for items like convergence, the dialog box shows different options. These could be point groups or sensor fields. Multiple point groups can be selected at once, but all the data will be written to the same file unless the filename definition had the field [Point Group] specified.
  3. Select the Network and Time interval, with optional Start and End dates.
  4. Click Display to load all available epochs for selection.
  5. Use the mouse right-click context menu to select start and end epochs.
  6. Click OK.

Exporting will start, and a message appears when it is finished, indicating how many files have been written.