Delta Watch: Import & Export module

Need to send processed Delta Watch data to a third party?

The Import & Export module can take care of that.

This guide explains the ease of use and flexibility that we put in your hands.

Log in or sign up for myTopcon to view this page:

Delta Watch: Import & Export module

Introduction

Delta Watch has got a highly user configurable Import & Export module. With this module it is possible to use a predefined export format or to define your own. The last is particularly useful when there is a need to provide third parties with a continuous processed data flow of all sorts.

The module can be found in the main access bar on the left by navigating to Interfaces > Import & Export.

In this module, you can add an unlimited number of import and export configurations, called tasks.

Each job can be configured to execute at specific time intervals and will run independent of other processing tasks. The custom format configuration is very flexible and allows you to configure things like:

  • Sensor type
  • Filename
  • Date/time format
  • Headers
  • Column separator
  • Data values
  • Units
  • Decimal places

The import capabilities of this module are used for items like:

  • ENV or THP sensor import (*.ENV or *.THP)
  • Delta Link control unit information (*.CUI)
  • Total station information file statistics (*.STNI)

The export options are endless and can be customized to fit the project specific needs.

Additionally, alarms can be configured to trigger events such as when no data has been sent or received for a certain amount of time. The Import & Export module also provides a status overview showing, for example, age of last action. The image below shows the status overview in the module. Similarly, but less detailed, info is available in Delta Live! on the dashboard. This will only show the last time data was imported or exported.

Create custom export formats

The Import & Export module has a few predefined export formats. These formats could be found in pre-2.15 versions of Delta Watch in the Automated networks module. The table below shows details on the Geodetic export formats.

Geodetic Data Type Export Formats
Export Type Replaces Export Option
GeodeticPointExport  
Kronos_3Ddeformation Export to information system with export format set to Kronos*
Kronos_Levelling (leveling section) Export to information system with export format set to Kronos*
Iris(Group; Name) Export to information system with export format set to Iris(Group; Name)*
Iris(Name; Group) Export to information system with export format set to Iris(Name; Group)*
ProcessedPointExport Option Processed Point Export

 

* The export type was normally set at Delta Watch > System > Automatic_processing > General >  Export_type.

 

Upgraded from a Delta Watch pre v3.x?

Make sure you have disabled ALL network exports before upgrading. Alternatively, log in to Delta Watch as Administrator, and navigate to Administration > System configuration > Support. Here, use the function to deactivate all exports from geodetic networks.

Most data export is done from automated networks, but if you were exporting from other modules, then you will find each format back in the export module once you select the Sensor type.

A format is a sort of template that tells what data should go in which columns, and, once defined, can be used by multiple jobs. Besides the predefined formats, you can also define your own format. 

  1. Navigate to Interfaces > Import & Export.
  2. Click the Custom format button in the top toolbar. A new dialog box, like the figure below, opens, showing you any existing formats.
  1. Click either Add Add or Duplicate Copy. To see an output example of the existing formats, select the item of interest, and the overview will update with the specific details.
  2. A dialog box opens where a new Custom format can be configured. The dialog box will look something like the image on the right. All fields need to be configured.
    • Name: Unique name for this format definition.
    • Filename extension: Currently only CSV export possible.
    • Sensor type: Selecting the correct type will ensure only applicable data fields are shown and that the definition only shows when creating an export for this sensor type. The following options are currently available:
      • Geodetic sensor
      • Meteorological
      • Structural monitoring
      • Convergence section
  • Import/Export: Defines the format type. Currently only Export can be selected.
  • File name and Timestamp format: Both these fields can be customized. Select an existing format using the drop-down menu, or add/edit a format. To add or edit, click Add format, and a dialog box will open where existing and new formats can be managed. When using Edit Edit or Add Add, a dialog box shows with all options that can be used, as per the image below. When done, be sure to select the format using the drop-down menu in the dialog box as per the figure above.
  • Column separator: Specify the character to use. Write [TAB] for the tab character.
  • Header configuration: If checked, then a header will be written as the first line of each data file. This can be an automatic header based on the chosen data columns or a custom header.
  • Column configuration:
    1. Add/remove columns as required. The action is always performed on the last column. Inserting or selective delete is not possible.
    2. For each column, where applicable, specify:
      • Column type
      • Unit
      • Decimal places
      • Text (only when fixed text is selected)
  1. Click OK to save the custom format configuration. This can now be used for automated or manual exports.

Configure automatic export

The export jobs are replacing all automated exports from pre Delta Watch v2.15. To create an export job, follow the steps:

  1. Add a new job definition by clicking on the Add button.
  1. A new dialog box will open where the new job can be defined.
    • Name: When naming tasks, keep in mind that you can only export from one project but can export multiple point groups at the same time!
    • Import/Export: Set to Export.
    • Sensor type: Current options are:
      • Geodetic sensor
      • Meteorological
      • Structural monitoring
      • Convergence section
      • Rail track
    • Export folder: Specify the destination folder. It is best to ensure a unique export folder per task.
  • Time interval, Start time, Export after processing: Together these fields specify when data export should happen. Check Extended settings to see all options. The Start time has two functions:
    • Set a date in the future so the job will not start until reached.
    • Specifying the time to control base time of the export. For example, setting the time to 00:10:00 and at 60 minutes Time interval, the job will export at 00:10:00; 01:10:00, 02:10:00, 03:10:00…….23:10:00.
  • Used export time zone: By default the timestamp of the exported data is in UTC. Change this field if the date/time stamp should reflect a different timezone.
  • Export type: A list with export formats. Any custom export formats for the selected sensor type will also be visible here.
  • Select project: Depending on which sensor type is exported, either a sensor-group or a sensor-field might need to be selected. If using groups, you can select multiple groups at once, but all the data will be written to the same file unless the filename definition has the [Point Group] field enabled.
  • Monitor data transfer: Complete this section to get a message on certain actions. The email server needs to be configured, and the project must have one or more Contacts. The following triggers are available:
    • Data transfer
    • Incorrect export
    • Confirmation (that data was exported)
  1. Click OK to save the new configuration, and then click Apply to write to the database.

Configure automatic import

Import configurations are used to import some of the special data files that Delta Link generates. These are the files that contain:

  • Basic weather station data (*.ENV or *.THP)
  • Control unit information (*.CUI)
  • Station information (*.STNI)

This article is limited to explaining the general setup of an Import job. For more details on each of the above items, there are specific articles which will explain the complete setup for each item.

Add a new job definition by clicking on the Add   button in the Import & Export module.

  • Name: Specify a name for this import task.
  • Import/Export: Set this to Import.
  • Sensor type: For Topcon Delta Watch only the below are applicable:
    • AMTS Import: Imports *.CUI and *.STNI
    • Meteorological: Imports *.ENV or *.THP
    • Extensometer: Imports custom Geoscope or CSV
    • Waterlevel: Imports custom Geoscope or CSV
    • Tilt meter: Imports Senceive, custom Geoscope, or VMT CSV format
  • Base directory: This is where the data will be imported from.
  • Time interval: To be able to alarm relatively fast on, for example, AC power loss of a Delta Link unit, ensure your import interval is set reasonably high, like 10 minutes or less, even though under normal conditions the CUI file is only generated hourly.

Complete the rest of the configuration. Available fields will change depending on the chosen sensor type. See the figure below for an example. When checking Extended settings, it is possible to specify items such as the timezone of data from the source file. Any data originating from Delta Link is always stored as UTC, so there is no need to change the setting.

 

Using the Manual export function

From this Import & Export module, we can also perform manual exports of data. It is possible to export data from a specific time in a specific (user-defined) format to a user-defined folder.

  1. In the Import & Export module, click on the Manual Export button in the main toolbar. See image.
  1. A new dialog box will open up. Make the configuration for the following fields:
    • Select project
    • Sensor type
    • Export type
    • Export folder
    • Used timezone
  2. Depending on the selected Sensor type and the current definitions in your DB, for items like convergence, the dialog box shows different options. These could be point groups or sensor fields. Multiple point groups can be selected at once, but all the data will be written to the same file unless the filename definition had the field [Point Group] specified.
  3. Select the Networks and Time interval, with optional Start and End dates.
  4. Click Display to load all available epochs for selection.
  5. Right-click to select start and end epochs.
  6. Click OK.

Exporting will start, and a message appears when it is finished, indicating how many files have been written.