Delta Watch has got a user configurable Import & Export module with lots of flexibility. With this module, as the name suggest, data can be imported and exported. Data export could use either preconfigured or a custom defined format. The last is particularly useful when there is a need to provide third parties with a continues flow of processed data of all sorts.
The module can be found under the Interface group in the Navigation bar and is called Import & Export, see image on right.
All data Import & Export tasks are created and configured in this module, and data flow monitoring can easily be enabled. Tasks can be scheduled to run at intervals or, for exports, directly after a calculation. These tasks will run independent of other processing tasks.
The custom-export-format allows for configuring items like:
- Dynamic filenames
- (Custom) Headers
- Date/time format
- Column separator
- Data value type; resolution and units
The IMPORT capabilities of this module are used for items like:
- Weather station sensor import (*.ENV or *.THP)
- Delta Link Control unit information (*.CUI, *.MDM)
- Total station information (*.STNI and *.BUB)
- Geotechnical sensors (Tilt, water level, inclinometer etc.)
The export options are endless and can be customized to fit the project specific needs.
Delta Watch also has a restful API interface that could be used to get data out of the DB instead of exporting files. Contact support for more details. |
Additionally, alarms can be configured to trigger events, such as when no data has been sent or received for a certain amount of time. The Import & Export module also provides a status overview showing, for example, age of last action. The image below shows the status overview in the module. Similar, but less detailed, info is available in Delta Live! on the dashboard. This will only show the last time data was imported or exported.
What is in this article
- The basic workflow.
- Configure Import task.
- Creating custom export format.
- Configure Automatic Export.
- Using the manual Export function.
- Troubleshooting.
The basic workflow
The Import & Export module is always used in conjunction with other modules, and because of that it should not be the first item to configure when creating a project or wanting to import data. Sensors will always need to be configured in their respective modules first before their data can be Imported or Exported. Details and extra instruction for module specific import tasks can be found in the help or How to documentation for that particular module. That leaves use with the following guidelines for a basic workflow.
- Require an active project, even if just importing system diagnostics information.
- Prepare and configure the module of interest.
- For import:
- Create the import task, make sure there are some data files.
- For Export:
REMEMBER: Delta Link data files always use UTC for any timestamp. |
Configure Import task
Import configurations are used to import data from one or more external sources at regular intervals, with options to monitor the data flow and raise alarm on specific values. This could be Delta Link specific data, or data from other geotechnical sensors. Below some examples.
Delta Link information | Geotechnical sensors |
|
|
This article is limited to explaining the general configuration of an Import task. For more details on each of the above items, there are separate articles that look at the specific requirement for a sensor. So, here is the general workflow to create an import task.
- Open the Import & Export module, found under the Interfaces group.
- Click on Add button to add a new task.
- Follow the order of the fields in the table below to complete the configuration.
- When completed: Close the configuration window and click Apply to save the changes.
For most sensors, only one import definition per project and/or import folder could to be created. This will automatically process specified compatible files. Separate import tasks only have to be configured if:
- Data arrives in separate folders.
- Different alarm limits should be applied.
When importing Delta Link system diagnostics files, it is strongly advised to create an import task per data-type per unit. This allows for much more flexibility in regard to alarming or when a particular Delta Link or Total Station will move to a different network or project altogether. |
Illustration below of import task configuration: A) Used for TS information, B) Used for geotechnical sensor data
Table with basic fields when configuring import task.
Overview of configurable fields for data import
Setting | Value |
---|---|
Name |
A user-chosen name for this import definition |
Import/Export |
Set to Import |
Sensor type |
Set to correct sensor type, see below table for more details. |
Base directory |
Directory where the data files can be found. |
Time interval |
At what interval to check for new data. If this is for CUI data and alarms are configured, then a relatively high interval of 5 or 10 minutes could be set. |
Import type |
Field changes based on selected sensor type above. See below table for extra details. Additional information can be found in sensor specific articles elsewhere on myTopcon. |
Select project |
The destination project for which this import is, not all Sensor type imports are project specific, see below table for details. |
Select Limit |
If a Limit was configured in the Limit Monitoring section of the Point and Limit management module, then select it here to be used. Data is checked upon import. |
Select sensor |
Depending on the imported Sensor type, specific sensors need to be selected. |
Sensor Type | Import type | Time interval | Selection |
---|---|---|---|
Control unit (info) | Control unit info (*.cui) | Selectable | Sensor only |
Modem log file (*.mdm) | |||
Total station (hardware info) | Total station info (*.stni) | Selectable | Sensor only |
Compensator (*.bub | |||
Inclinometer | Simple csv (*.csv) | Selectable | Project+Sensor |
Extensometer | Geoscope customized | Selectable | Project+Sensor |
Comma separated values (*.csv) | |||
Water level | Geoscope customized | Selectable | Project+Sensor |
Comma separated values (*.csv) | |||
Tilt meter | Geoscope customized | Selectable | Project+Sensor |
Senceive (*.csv) | |||
VMT data format (*.csv) | |||
Strain gauges | Geoscope customized | Selectable | Project+Sensor |
VMT data format (*.csv) | |||
Meteorological | Meteorological (*.env) | Fixed 10 minutes | Sensor only |
Levelling data | Simple csv (*.csv) | Selectable | Project+Sensor |
Sixense 1D [in] (*.txt) |
Extended settings (optional)
These settings allow a little more control and configuration of the data, especially if the data is older data or created with local timestamps instead of the standard expected UTC time.
Setting | Value |
---|---|
Active | allows activating/deactivating this import task |
Start time | Specify the time for the start of the data. Normally this is in the future but could be a time in the past matching the date/time from which point onwards the data needs to be processed. Any data from before this time will not be processed. |
Data time zone | If the timestamp in the data files is NOT in the UTC time zone, then specify here the correct time zone so that Delta Watch can convert the timestamp information correctly. By default, Delta Watch assumes any timestamp is in UTC time zone. |
Monitor data transfer (optional)
Configure to get an email message triggered by the data flow. For example: An email could be sent when data has not arrived for a certain amount of time. Click on to Add one of the project recipients and define the amount of time after which the message should go out. If the list of recipients is empty, then configure these first in the Contacts section of the Project module.
Finished
Once all is configured and saved to the database, Delta Watch might directly start with the import, depending on the start time and interval settings. The Import status can always be seen in this module in the columns Latest datatransfer and Total datatransfer. Below figure shows an example of this.
Creating Custom export format
The import and Export module allow for easy creation of so called flat-files, or CSV files. A configuration is made for a specific sensor type, and one or more of the following can be configured:
- Filename layout
- Multiple custom date and timestamp formats
- Custom order of columns
- Automatic or custom header
- Export units
- Resolution of values
- Fixed text fields can be added
- Column and decimal separators
There are a few pre-defined export formats available, these are mostly used for backwards compatibility with other systems that require a very specific format.
Section A) shows how to configure the file format, and section B) shows how to define custom filename and timestamp format strings.
Upgraded from a Delta Watch pre v3.x? Make sure you have disabled ALL network exports before upgrading! Alternatively, login to Delta Watch as Administrator and navigate to the System Configuration section. Under the Support section, use the function to Deactivate all exports from geodetic networks. Most data export was done from Automated networks, but if you were exporting from other modules, then you will find each format back in the export module once you select the Sensor type. |
A. Configure the format
A format definition can be seen as a template that defines what data should go in which columns, the type of units and number of decimal places that should be used. A format definition can be used by multiple export tasks. This section shows how to define a Custom format.
- Open the Import & Export module, found in the Interfaces group.
- Click the Custom Format button in the toolbar. A new dialog window opens, like the illustration below on the left, showing all current available custom formats.
Want to see an output example? Select the item of interest under Configurations and the overview will update with the specific details. Click on Preview to generate a dummy file. |
- Click either Add or Duplicate
- A configuration screen opens where the new format can be configured. This will look something like the in the above illustration on the right. All fields need to be configured; the below table gives further details.
- Click OK to save the configuration. Now it can be used for any exports.
Settings | Details |
---|---|
A user-chosen name for this format definition. Currently only .CSV export possible. Selecting the correct type will ensure only applicable data fields are shown and that the definition only shows when creating an export for this sensor type. The following options are currently available:
Defines the format type. Currently only Export can be selected. Select an existing formatting using the drop-down control, or add/edit a format. When clicking on Edit or Add an editor will show, as per image. Specify the character to use. Type [TAB] for the tab character Select dot or comma Select an existing formatting using the drop-down control, or add/edit a format. When clicking on Edit or Add an editor will show, as per image. Timestamp format can now be defined using standard ISO codes. If enabled, then a header will be written as the first line of each data file. This can be an automatic header based on the chosen data columns or a custom header. When creating a custom header, do not forget to include the column separator and unit indicators.
By default, only 5 columns will show, and the first one will always be the timestamp.
|
Configure Automatic EXPORT
The Export tasks are replacing all automated exports from pre Delta Watch v2.15 versions. |
Once the module that is processing the source data is configured, and preferably has already done some processing, it is time to create the export task. Follow the below steps to do so.
- Add a new task definition by clicking on the Add. See below illustration
- A configuration form will show where the new tasks can be defined. The below illustration shows 2 typical configuration examples, and the table below it clarifies the available fields.
- Click OK to save the new configuration, then click Apply to write to the database.
Illustration left: Example of Geodetic data export, performed directly after an epoch calculation.
Illustration right: Export definition example for (any) geotechnical sensor, exported at hourly intervals.
Setting | Details |
---|---|
Memorable name for this task. When naming tasks keep in mind that a task can only export from one project but can export multiple Point groups at the same time! Set to Export The following options are currently available:
Specify the destination folder. Ensure a unique export folder per task so that data is not accidentally mixed or overwritten. Together, these fields specify when data export should happen.
Export after processing is available for some export types and, when enabled, the Time Interval field is ignored, and export is triggered by the respective module after processing new data.
By default, the timestamp of the exported data is in UTC. Change this field if the date/time stamp should reflect a different timezone. A list with export formats. Any custom export formats for the selected sensor type will also be visible here. Select from which project the data will be exported Depending on which sensor type is exported, either a sensor-group or a sensor-field might need to be selected. Multiple entries can be selected, but if selecting point groups all the data will be written to the same file unless the filename definition has the [Point Group] field enabled. Otherwise this can give unwanted results, like duplicate entries. |
Complete this section to get a message on certain actions.
The mail server needs to be configured and the project must have one or more Contacts.
The following triggers are available:
- Data transfer
- Incorrect Export
- Confirmation (that data was exported)
Using Manual Export function
With manual exports, the user can perform a one-time export while defining things such as the time frame and the specific (user-defined) format. Or just re-export a certain dataset.
- Click the Manual Export button in the toolbar of the Import & Export module, see below illustration.
- A configuration window opens. Make the configuration for the following fields:
- Select Project
- Sensor type
- Export type
- Export folder
- Used Timezone
(Timestamp of the Exported data will be in this timezone)
- Depending on the selected Sensor type and the current definitions in your DB, for items like convergence, the dialog box shows different options. These could be point groups or sensor fields. Multiple point-groups can be select at once, but all the data will be written to the same file, unless the filename definition had the field [Point Group].
- Select the Network and Time Interval, different fields will show depending on the selection here.
- Configure the options and click on Display to load all available epochs for selection.
- Use the mouse right-click context menu to select the Start and End epochs.
- Click OK.
Exporting will start, when completed a message will show indicating how many files have been written.
Troubleshooting - Data Import
All troubleshooting for Imports will be because there appears to be No data. Below some points to check:
- Age of last data transfer for each task as shown in the Import and Export module.
- Very recent...
If not long ago, then the data file might have been empty, or charts are not scaled correctly when attempting to view the data. Locate the file in the pro-ject Archive folder and check its content. - Long ago...
More problematic, could be anything that stops data flow from Delta Link turned OFF to FTP server not running, or no internet connection.
- Very recent...
- Check the message archive and look for import failures, how?
Open the Message archive and check if there are any messages relating to data import. If there were incompatible, faulty or corrupted files, then a warning could be found here. Depending on the last import more than 100 messages need to be loaded. Make sure to check and use the column filters.
- Are the data files waiting in the import folder:
- Yes...
- If this is the first time to import, be patient and let Delta Watch execute the task as per time interval setting.
- Is the import task set to Active?
- Does the import task points at the right import folder.
- Is the Start date equal or older than the date of the first data file.
- Depending on import type, is the correct project and/or sensor selected
- Is the filename correct, for example, no uppercase file extensions!
- Does the file have actual data?
- Check the file format and sensor names, use a previously accepted file from the archive to compare with if possible.
- Is the file corrupt, like unexpected and of file.
- No...
- Check the data delivery or, in case of a Delta Link, attempt to connect to it using the web-portal. If no connection can be made, then a site visit to investigate is required. Check the last records from the CUI file to see if the power source might have starting to fail.
- Files might have been archived but not processed. Most likely reason is a change in name or other details somewhere. Check the full filename, text is case sensitive.
- Yes...
Troubleshooting - Data Export
Troubleshooting for data Exports can be narrowed down to one of the following:
- No (new) data
- No data to calculate an epoch.
- Epoch calculation failed.
- Check age of last export, if not recent but epochs exist, then might point to missing reference points or other epoch calculation failure.
- Destination folder was moved/renamed/deleted
- External drive is not available
- User had no rights to write a file
- Export task checks
- was de-activated
- Has a start time in the future
- Filename definition is wrong, i.e. each export overwrites the same file
- When exporting Geodectic data
- Point group deleted or adjusted.
- Points in the point group were not calculated this time.
- Local coordinate system wrongly created or assigned to the point group
- Export did happen, accoording to message archive
- File was already consumed by another program (and failed processing)
- File was accessed by another program while still being written