Delta Watch has got a highly user configurable Import & Export module. With this module, it is possible to use a predefined export format or to define your own. The last is particularly useful when there is a need to provide third parties with a continuous processed data flow of all sorts.
The module can be found in the main access bar on the left by navigating to Interfaces > Import & Export.
In this module, you can add an unlimited number of import and export configurations, called tasks.
Each job can be configured to execute at specific time intervals and will run independent of other processing tasks. The custom format configuration is very flexible and allows you to configure things like:
- Sensor type
- Date/time format
- Column separator
- Data values
- Decimal places
The import capabilities of this module are used for items like:
- ENV or THP sensor import (*.ENV or *.THP)
- Delta Link control unit information (*.CUI)
- Total station information file statistics (*.STNI)
- Geotechnical sensors (Tilt, water level, inclinometer, etc.)
The export options are endless and can be customized to fit the project specific needs.
Additionally, alarms can be configured to trigger events, such as when no data has been sent or received for a certain amount of time. The Import & Export module also provides a status overview showing, for example, age of last action. The image below shows the status overview in the module. Similar, but less detailed, info is available in Delta Live! on the dashboard. This will only show the last time data was imported or exported.
Create custom export formats
The Import & Export module has a few predefined export formats. In Delta Watch versions older than v2.15, the export was done from within the Automated networks module. That method is now obsolete, and the new module should be used. The table below shows details on the geodetic export formats, their new name, and what they were called in the older versions.
|Export Type||Replaces Export Option (pre-2.15 versions)|
|Kronos_3Ddeformation||Export to information system with export format set to Kronos*|
|Kronos_Levelling||(leveling section) Export to information system with export format set to Kronos*|
|Iris(Group; Name)||Export to information system with export format set to Iris(Group; Name)*|
|Iris(Name; Group)||Export to information system with export format set to Iris(Name; Group)*|
|ProcessedPointExport||Option Processed Point Export|
* The export type was normally set at Delta Watch > System > Automatic_processing > General > Export_type.
Upgraded from a Delta Watch pre v3.x?
Make sure you have disabled ALL network exports before upgrading. Alternatively, log in to Delta Watch as Administrator, and navigate to Administration > System configuration > Support. Here, use the function to deactivate all exports from geodetic networks.
Most data export was done from Automated networks, but if you were exporting from other modules, then you will find each format back in the export module once you select the Sensor type.
Configure the format
A format is a sort of template that tells what data should go in which columns, and, once defined, can be used by multiple jobs. Besides the predefined formats, users can now define their own format. Follow the simple steps below for how to do this.
- Navigate to Interfaces > Import & Export module
- Click the Custom format button in the top toolbar. A new dialog box, like the figure below, opens, showing all current available formats.
- Click either Add or Duplicate . To see an output example of the existing formats, select the item of interest, and the overview will update with the specific details.
- A dialog box opens where a new Custom format can be configured. The dialog box will look something like the image on the right. All fields need to be configured. The table gives some details here.
- Click OK to save the custom format configuration. This can now be used for automated or manual exports.
|Name||A user-chosen name for this format definition|
|Filename extension||Currently only .CSV export possible|
Selecting the correct type will ensure only applicable data fields are shown and that the definition only shows when creating an export for this sensor type. The following options are currently available:
Defines the format type. Currently only Export can be selected.
|File name||The filename format, which can be customized. See below for more details on options available.|
Specify the character to use. Enter [TAB] for the tab character.
Select either dot or comma to be used.
|Timestamp format||The timestamp format to be used, which can be customized. See below for more details.|
If enabled, then a header will be written as the first line of each data file. This can be an automatic header based on the chosen data columns or a custom header.
File name and timestamp formats
Select an existing formatting using the drop-down control or add/edit a format. To add or edit, click on Add format, and a dialog box will open where existing and new formats can be managed. When using Edit or Add , a dialog box shows with all options that can be used, as per the image below. When done, be sure to select the format using the drop-down control as in the above image.
Configure automatic export
The export jobs are replacing all automated exports from pre Delta Watch v2.15 versions. To create an export job, follow the below steps:
- Add a new job definition by clicking on the Add button.
- A dialog box will open where the new task can be defined. The table below, Export Task - Configurable Fields, clarifies the available fields, and the below image shows two typical configuration examples.
- Click OK to save the new configuration, and then click Apply to write to the database.
|Name||Memorable name for this export. When naming tasks, keep in mind that you can only export from one project but can export multiple Point groups at the same time!|
|Import/Export||Set to Export.|
The following options are currently available:
Specify the destination folder. Ensure a unique export folder per task so that data is not accidentally mixed or overwritten.
Together these fields specify when data export should happen. Enable Extended settings to see all options. The Start time has two functions:
Export after processing is available for some export types and, when enabled, the Time interval field is ignored, and export is triggered by the respective module after processing new data.
|Used export time zone||By default, the timestamp of the exported data is in UTC. Change this field if the date/time stamp should reflect a different timezone.|
|Export type||A list with export formats. Any custom export formats for the selected sensor type will also be visible here.|
Select from which project the data will be exported
Depending on which sensor type is exported, either a sensor-group or a sensor-field might need to be selected. Multiple entries can be selected, but if selecting point groups, all the data will be written to the same file unless the filename definition has the [Point Group] field enabled. Otherwise this can give unwanted results, like duplicate entries.
|Monitor data transfer||
Complete this section to get a message on certain actions. The email server needs to be configured, and the project must have one or more Contacts. The following triggers are available:
Configure automatic import
Import configurations are used to import external data. This could be the special data files that Delta Link generates, such as:
- Basic weather station data (*.ENV or *.THP)
- Control unit information (*.CUI)
- Station information (*.STNI)
Or data files from other geotechnical sensors, such as below but not limited to:
- Tilt meters
- Water level sensors
This article is limited to explaining the general setup of an Import task. For more details on each of the above items, there are specific articles which will explain the complete setup for each item.
- Add a new job definition by clicking on the Add button in the Import & Export module. A dialog box shows similar as for exporting.
- Follow the order of the fields in the below table to complete the configuration.
- When completed, close the dialog box, and click Apply to save the changes.
|For most sensors only one import definition per project and/or import folder needs to be created. This will automatically process all compatible files in the specified import folder.|
|Name||A user-chosen name for this format definition|
|Import/Export||Set to Import|
|Sensor type||Set to correct sensor type|
Directory where the data files can be found
How often Delta Watch should check for new data. If this is for CUI data and alarms are configured, then a relatively high interval of 5 or 10 minutes might need to be set.
Depending on the sensor type, there might be more than one format available. Make sure you select the correct format. More info could be found in sensor-specific articles elsewhere.
Current available imports are:
|Select project||The destination project for which this import is|
If alarms are configured for any sensor(s) that are imported by this task, then select here who should receive the emails. One or more project contacts would need to be defined to be able to select them.
Allows the user to enter some information here
Extended settings (optional)
These settings allow a little more control and configuration of the data, especially if the data is old data or created with local timestamps instead of the standard expected UTC time.
Allows activating/deactivating this import definition
|Start time||Specify the time for the start of the data. Normally this is in the future but could be a time in the past matching the date/time from which point onwards the data needs to be processed. Any data from before this time will not be processed.|
|Data time zone||If the timestamp in the data files is NOT in the UTC time zone, then specify here the correct time zone so that Delta Watch can convert the timestamp information correctly. By default, Delta Watch assumes the data is in UTC time zone.|
Monitor data transfer (optional)
It is possible to configure email messages triggered by the data import definition. For example, an email could be sent when data has not arrived for a certain amount of time. This can be configured here. Click on to Add one of the project recipients and define the amount of time after which the message should go out. If the list of recipients is empty, then configure these first in the Project > Contacts section.
Once all is set up and saved to the database, Delta Watch might, depending on the start time and interval settings, already start with the import. The Import status can always be seen in this module in the columns Latest datatransfer and Total datatransfer. An example of this can be seen in the figure below.
Using the Manual export function
From this Import & Export module, you can also perform manual exports of data. It is possible to export data from a specific time in a specific (user-defined) format to a user-defined folder.
- In the Import & Export module, click on the Manual export button in the main toolbar. See image.
- A new dialog box will open up. Make the configuration for the following fields:
- Select project
- Sensor type
- Export type
- Export folder
- Used timezone
- Depending on the selected Sensor type and the current definitions in your database, for items like convergence, the dialog box shows different options. These could be point groups or sensor fields. Multiple point groups can be selected at once, but all the data will be written to the same file unless the filename definition had the field [Point Group] specified.
- Select the Network and Time interval, with optional Start and End dates.
- Click Display to load all available epochs for selection.
- Use the mouse right-click context menu to select start and end epochs.
- Click OK.
Exporting will start, and a message appears when it is finished, indicating how many files have been written.