Data Source Editor

The Data Source Editor Tool

The Data Source Editor tool is introduced with inorigo 4.8. It enables users to define and save connections to- and configurations of data sources. These data sources can then be used throughout the inorigo suite and external tools via the API. The Data Source Editor provides essential capabilities for data integration in inorigo.

Search Field

Use the Search Field located at the top of the Sidebar to search with contains for any Data Source or Connection.

 

Tag

The  displays the type of Connection / Data Source

 

 

Usage

Used X time(s) displays the number of usages for a Connection / Data Source. Select an item and press Used X times in the top-right corner to see the usages.

If a Connection / Data Source is in use it cannot be deleted. To delete a Connection / Data Source in use – first find and delete or clear the usages.

Status

 OK – The Connection / Data Source works is available.

 Pending – inorigo is waiting for the Connection / Data Source to respond.

A Connection / Read Timeout can be configured from the Connection settings.

 Fail – The Connection / Data Source could not be established. Hover the Fail icon to display additional information

Right-Click Menu

Open in new tab Opens Data Source/Connection in new tab
Open in Data Source/Connection Editor Opens Data Source/Connection
Create copy Opens a Copy of the selected Data Source or Connection with the given name Copy of [Unit Name]. Notice that the copy is not saved to the database until you press
Copy ID Copies the ID of the Data Source/Connection
Bookmark Data Source/Connection Adds Data Source/Connection to the Library

Connections

Create a Connection to set up a type and a location, such as an SQL server… or a Folder… Once set, Data Sources can make use of a connection. Create a Data Source to set up what content from a location that is to be included in the data source. Once the Data Source has been saved it will be available for use throughout the inorigo software.

You can use the same connection for all data sources that share a location and type. Say that you have an integration folder with multiple files that are to be used as Data Sources, using the same connection for all data sources is fast and easy to update if the integration folder is relocated. You can always change what connection a data source make use of later on. Here I have made use of the same connection for these two sources.

Create Connection

Simply press Create Connection in the left sidebar menu under the tab Connections to create a new connection. Once it has been created it will be available from the list of connections in the left sidebar.

Edit / Delete Connection

Once a connection has been created it can be opened and edited by selecting it from the left sidebar menu. Make your changes and then press Save Changes to apply them. Leaving the page without pressing Save, will not apply any changes.

A connection in use cannot be deleted. To delete a connection first delete or redirect the data sources that use the connection. Click Usages: 

in the top-right corner displays where a connection is being used.

It’s possible to click the steps in the wizard to jump through steps when editing a Connection

Connection Types

Folder

 Create a connection to a Folder Path. Files located within the folder can be used when a data source is established.

 

 

Name (required) Name the connection to find and use it in Data Sources and the Data Source Editor
Connection Type (required) Choose the type of connection. Once a type is chosen, Type specific settings are revealed.
Folder (required) Enter a path to the folder

The default folder path is set to the inorigo integration folder:

C:\Program Files \Apache Foundation\inorigoXYZ\inorigo\integration

User Password Enter a User name and Password to access the given folder path.

For paths where the inorigo instance has access rights, no User or Password is required.

Notice that its the inorigo-user running the server that requires access, not you the user setting up the Connection. By default, inorigo has access to the integration folder.

Test Connection Test if the connection is working.

Notice that test connection only tests the path, not user authentication for the data source.

Create Connection / Save Changes Create the connection to save it to the Data Source Editor. Once a connection has been created it can be opened and edited at any time.

URL

Name (required) Name the connection to find and use it in Data Sources and the Data Source Editor
Connection Type (required) Choose the type of connection. Once a type is chosen, Type specific settings are revealed.
Base URL (required) Enter a Base URL
User Password Enter a User name and Password to access the given folder URL.

For URLs where the inorigo instance has access rights, no User or Password is required.

Notice that its the inorigo-user running the server that requires access, not you the user setting up the Connection. By default, inorigo has access to the integration folder.

Test Connection Test if the connection is working.

Notice that test connection only tests the path, not user authentication for the data source.

Create Connection / Save Changes Create the connection to save it to the Data Source Editor. Once a connection has been created it can be opened and edited at any time.

 

 

Advanced

 

Connection Timeout (required) Set a time (in seconds) to attempt to establish a connection. The connection will fail when if it has not been established when the given timeout has been exceeded.

Default: 20 seconds

Read Timeout (required) Set a timeout (in seconds) for a data source to be loaded from the connection. The connection will fail if the data sources has not been loaded when the given timeout has been exceeded.

Default: 20 seconds

Request Headers: Key Request Headers: Value HTTP header fields are components of the header section of request and response messages. Use the request Key and Value fields to define a header field name and value to request.

Example:

Key: Content-Length

Value: 348

More information about header fields can be found here

 

SQL Server

Name (required) Name the connection to find and use it in Data Sources and the Data Source Editor
Connection Type (required) Choose the type of connection. Once a type is chosen, Type specific settings are revealed.
Host (required) Port Enter the server address and if required a port. Example: 192.168.0.1:144

Default port is 143

Integrated Security (required) When yes inorigo will use the account that the service is running under to authorize the connection to the host.

When no two input fields appear to provide a username and password.

User (required) Password (required) If integrated security is disabled,

Enter a User name and Password to access the given SQL Server.

 Test Connection Can be used to determine if access is granted.
Database (required) Choose the database from the list of available databases from the server.
Test Connection Test if the connection is working.

The test button require login credentials to work for SQL types, unless  integrated security is used.

Create Connection / Save Changes Create the connection to save it to the Data Source Editor. Once a connection has been created it can be opened and edited at any time.

 

Advanced

 

Instance Enter the name of the SQL-instance if there are many instances.
Extra Parameters: Key Extra Parameters: Value Add extra parameters to the connection string. 

 

Data Sources

Data Sources can make use of a connection. Create a Data Source to set up what content from a location that is to be saved as a Data Source. Once saved, a Data Source is available for use in inorigo.

Create Data Source

Simply press Create Data Source in the left sidebar menu under the tab Data Sources to create a new Data Source.

A 4-step wizard appears where a Name and Data Source Type is first chosen. The choice of Data Source Type and Connection in step 1 and 2 of the wizard determine what settings that are available for step 3 and 4.

Information for each of the available Data Sources can be found here.

Information for each of the available types of Connections can be found here.

Once it has been created it will be available from the list of Data Sources in the left sidebar, in Knowledge Sets under External from the Data Type menu.

Edit / Delete Data Source

Once a Data Source has been created it can be opened and edited by selecting it from the left sidebar menu. Make your changes and then press Save Changes to apply them. Leaving the page without pressing Save, will not apply any changes.

Any changes to a Data Source may have immediate effects on modules or tools where it used. Be sure to confirm updates to a used Data Source does lead to any unwanted changes with any of its usages.

A Data Source in use cannot be deleted. To delete a connection first delete or redirect the data sources that use the connection. Click Usages  in the top-right corner displays where a connection is being used.

It’s possible to click the steps in the wizard to jump through steps when editing a Data Source

Data Source Types

CSV

1. Name & Type

Data Source Name (required) Name the Data Source to find and use it in Data Source Editor and throughout inorigo.
Data Source Type (required) Choose the type of Data Source to create. Once a type is chosen, type-specific settings are in the next step of the wizard.

 

 

2. Connection

Choose an available connection for the selected Data Source Type. If the desired connection has not yet been created, press Create New Connection.

Read more about the Connections types and settings here.

 

 

3.A – Based on a Folder Connection

Target (required) Target either a single file by typing in Filename or target many files by typing in a Filename pattern (using Windows Wildcard symbols – more information here.

When multiple files are targeted they are required to have the same Delimiter and Header row

Source Character Set Set the character set that the source file is encoded in. Default character set is UTF-8

 

Delimiter (required) Choose which symbol that separates the data in the CSV-file(s) into columns.

·         Comma (,)

·         Semicolon (;)

·         Space ( )

·         Tab ( )

·         Other Delimiter reveals a new field where a custom delimiter can be set.

Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

 

3.B – Based on a URL Connection

 

Method (required) Choose HTTP Request method. Chosen method works as a request-response protocol between the URL and inorigo. You can read more about HTTP Request methods here

Typical: POST or GET

Default: GET

SOAP Action Specifies the SOAP (Simple Object Access Protocol) action to use in the HTTP header when invoking this method as a web method via HTTP
Accept (required) The Accept request-header field is used to specify file types that are acceptable for the response.

Typical: Application/xml Application/json

Default: text/xml

Content-Type (required) Default: text/xml; charset=utf-8
Body Request body if Method is set to POST
Query  String Key Query String Value Enter additional parameters to the URL address to assign specific values. ??
Target (required) Enter the path to the target file. Target is added to the path defined by the connection. The full given path is displayed underneath the Target field.
Source Character Set Set the character set that the source file is encoded in.

Defaults to: Windows 1251

Delimiter (required) Choose which symbol that separates the data in the CSV-file(s) into columns.

·         Comma (,)

·         Semicolon (;)

·         Space ( )

·         Tab ( )

Other Delimiter reveals a new field where a custom delimiter can be set.

Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

4. Columns

Columns has the same settings for every type of data source, you can find them here

Excel File

Applicable Formats

Tested and supported formats are: XLS, XLSX and XLXM. Although not tested, it is likely that other formats based on Apache POI such as Open Office also works.

1. Name & Type

Data Source Name (required) Name the Data Source to find and use it in Data Source Editor and throughout inorigo.
Data Source Type (required) Choose the type of Data Source to create. Once a type is chosen, type-specific settings are in the next step of the wizard.

 

2. Connection

Choose an available connection for the selected Data Source Type. If the desired connection has not yet been created, press Create New Connection.

Read more about the Connections types and settings here.

 

3.A – Based on a Folder Connection

Target (required) Target either a single file by typing in Filename or target many files by typing in a Filename pattern (using Windows Wildcard symbols – more information here.

The full path as it is defined in the Connection and the Target field is displayed under the entry field.

When multiple files are targeted they are required to use the same sheet number, header row and first data row.

Source Character Set Set the character set that the source file is encoded in.

Defaults to Windows 1251

Sheet number (required) ·         Enter the Excel sheet number. It is not possible to read from more than one sheet per Data Source.

·         Default: 0

Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

3.B – Based on a URL Connection

Method (required) Choose HTTP Request method. Chosen method works as a request-response protocol between the URL and inorigo. You can read more about HTTP Request methods here

Typical: POST or GET

Default: GET

SOAP Action Specifies the SOAP (Simple Object Access Protocol) action to use in the HTTP header when invoking this method as a web method via HTTP
Accept (required) The Accept request-header field is used to specify file types that are acceptable for the response.

Typical: Application/xml Application/json

Default: text/xml

Content-Type (required) Default: text/xml; charset=utf-8
Body Request body if Method is set to POST
Query  String Key Query String Value Enter additional parameters to the URL address to assign specific values. ??
Target (required) Enter the path to the target file. Target is added to the path defined by the connection. The full given path is displayed underneath the Target field.
Source Character Set Set the character set that the source file is encoded in.

Defaults to: Windows 1251

Sheet Number (required) ·         Enter the Excel sheet number. It is not possible to read from more than one sheet in per Data Source.

Default: 0

Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

4. Columns


Columns has the same settings for every type of data source, you can find them here

HTML Table

Format

HTML needs to be Well Formed, meaning that it must adhere to the syntax rules specified by the XML 1.0 specification in that it must satisfy both physical and logical structures. Read more about Well Formed documents here

1. Name & Type

Data Source Name (required) Name the Data Source to find and use it in Data Source Editor and throughout inorigo.
Data Source Type (required) Choose the type of Data Source to create. Once a type is chosen, type-specific settings are in the next step of the wizard.

 

 

2. Connection

Choose an available connection for the selected Data Source Type. If the desired connection has not yet been created, press Create New Connection.

Read more about the Connections types and settings here.

 

3.A – Based on a Folder Connection

Target (required) Target either a single file by typing in Filename or target many files by typing in a Filename pattern (using Windows Wildcard symbols – more information here.

The full path as it is defined in the Connection and the Target field is displayed under the entry field.

When multiple files are targeted they are required to have the same Header Row and First Data row

Source Character Set Set the character set that the source file is encoded in.

Defaults to Windows 1251

Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

3.B – Based on a URL Connection

Method (required) Choose HTTP Request method. Chosen method works as a request-response protocol between the URL and inorigo. You can read more about HTTP Request methods here

Typical: POST or GET

Default: GET

SOAP Action Specifies the SOAP (Simple Object Access Protocol) action to use in the HTTP header when invoking this method as a web method via HTTP
Accept (required) The Accept request-header field is used to specify file types that are acceptable for the response.

Typical: Application/xml Application/json

Default: text/xml

Content-Type (required) ??

Default: text/xml; charset=utf-8

Body Request body if Method is set to POST
Query  String Key Query String Value Enter additional parameters to the URL address to assign specific values. ??
Target (required) Enter the path to the target file. Target is added to the path defined by the connection. The full given path is displayed underneath the Target field.
Source Character Set Set the character set that the source file is encoded in.

Defaults to: Windows 1251

Sheet Number (required) ·         Enter the Excel sheet number. It is not possible to read from more than one sheet in per Data Source.

Default: 0

Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

4. Columns

Columns has the same settings for every type of data source, you can find them here

SQL

1. Name & Type

Data Source Name (required) Name the Data Source to find and use it in Data Source Editor and throughout inorigo.
Data Source Type (required) Choose the type of Data Source to create. Once a type is chosen, type-specific settings are in the next step of the wizard.

 

2. Connection

Choose an available connection for the selected Data Source Type. If the desired connection has not yet been created, press Create New Connection.

Read more about the Connections types and settings here.

 

3. Data Source

Mode (required) Table:

If selected – all available tables will be presented in a drop down list.

 

 

Statement: If selected – A statement entry field will be displayed. Use it to enter an SQL statement to define what to include in the Data Source

 

4. Columns


Columns has the same settings for every type of data source, you can find them here

XML

Format

XML needs to be Well Formed, meaning that it must adhere to the syntax rules specified by the XML 1.0 specification in that it must satisfy both physical and logical structures. Read more about Well Formed documents here

1. Name & Type

Data Source Name (required) Name the Data Source to find and use it in Data Source Editor and throughout inorigo.
Data Source Type (required) Choose the type of Data Source to create. Once a type is chosen, type-specific settings are in the next step of the wizard.

 

2. Connection

Choose an available connection for the selected Data Source Type. If the desired connection has not yet been created, press Create New Connection.

Read more about the Connections types and settings here.

 

3.A Based on a Folder Connection

Target (required) Target either a single file by typing in Filename or target many files by typing in a Filename pattern (using Windows Wildcard symbols – more information here.

The full path as it is defined in the Connection and the Target field is displayed under the entry field.

When Multiple files are targeted in one data source they are required to follow the same configuration settings

Source Character Set Set the character set that the source file is encoded in.

Defaults to Windows 1251

Parse Mode XPath uses path expressions to select nodes or node-sets in an XML document. Choose a Parse Mode to reveal additional entry fields.
Row Path

(if Parse Mode = XPath) (required)

Enter a path in the following structure “/data/people/person”.

Give a source for every column as a continuous search path starting for each row.

If you set name as a column its possible to differentiate name as either an attribute for Person or its own attribute in Person:

<data><People><Person><name>Alva</name></Person><Person name=”Amy“></Person></People></data>”

Default: /TR

Row Tag

(if Parse Mode = Table) (required)

Enter a tag that identifies a row in the XML sheet.

Example: TR

Column Tag (required) Enter a tag that identifies a column in the XML sheet.

Example: TD

Table Path Use to set what table to include if the XML contains more than one table. All tables are included if the entry field is left blank.
Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

If the given Column tag is TD inorigo will automatically look for TH to find table headers.

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

Advanced

XSLT Enter XSLT to transform the structure of the XML document before its read by inorigo.

This can be used, for example, to change the order of the content, to add or remove information, or selecting selected parts of the basic document.

 

 

3.B – Based on a URL Connection

Method (required)

Add more

Choose http Request method. Chosen method works as a request-response protocol between the URL and inorigo.

Default: GET

SOAP Action

Add more

Specifies the SOAP (Simple Object Access Protocol) action to use in the HTTP header when invoking this method as a web method via HTTP
Accept (required)

Add more

The Accept request-header field is used to specify file types that are acceptable for the response.

Default: text/xml

Content-Type (required) Charset for the content

Default: text/xml; charset=utf-8

Body Request body if Method is set to POST
Query  String Key Query String Value Enter additional parameters to the URL address to assign specific values. ??
Target (required) Enter the path to the target file. Target is added to the path defined by the connection. The full given path is displayed underneath the Target field.
Source Character Set Set the character set that the source file is encoded in.

Defaults to: Windows 1251

Parse Mode XPath uses path expressions to select nodes or node-sets in an XML document. Choose a Parse Mode to reveal additional entry fields.
Row Path

(if Parse Mode = XPath) (required)

Enter a path in the following structure “/data/people/person”.

Give a source for every column as a continuous search path starting for each row.

If you set name as a column its possible to differentiate name as either an attribute for Person or its own attribute in Person:

<data><People><Person><name>Alva</name></Person><Person name=”Amy“></Person></People></data>”

Default: /TR

Row Tag

(if Parse Mode = Table) (required)

Enter a tag that identifies a row in the XML sheet.

Example: TR

Column Tag (required) Enter a tag that identifies a column in the XML sheet.

Example: TD

Table Path Use to set what table to include if the XML contains more than one table. All tables are included if the entry field is left blank.
Header Row Enter the number of the row where the Header are placed in the files. The first row in the file is written as zero in the input field. If no headers are available in the file/files headers will be automatically be named as: #1, #2, #3, #4

Notice! If the given Column tag is TD inorigo will automatically look for TH to find table headers.

Default: 0

First Data Row (required) Enter the row number for the first data row to be included in the Data Source, so that inorigo will know on what row it should start reading the data.

Default: 1 (Header is 0)

Last Data Row Set a last row to include in the Data Source – rows below will not be read and will not be available in the data source.

Only set a last data row if you do not want to include all rows below the first data row.

Leaving the field blank will automatically include all rows (including rows that are continuously added to the source file) below the first data row.

 

Advanced

XSLT Enter XSLT to transform the structure of the XML document before its read by inorigo.

This can be used, for example, to change the order of the content, to add or remove information, or selecting selected parts of the basic document.

 

4. Columns


Columns has the same settings for every type of data source, you can find them here

 

Data Source Columns

Columns

Columns allows you to

  1. Choose what columns to include
  2. Format Names

Press Add Columns to reveal all columns.

If the columns do not load as intended at this stage, make sure that the connection and the Data Source is properly configured.

The Add Columns Pop-up displays all columns with the Name, Data Type and order given in the source.

You can make changes to how inorigo interpret and display these columns in the Data Source – but can always come back to the Add Columns Pop-up to view the columns as they are defined in the source.

Columns as they are defined by the source

Tick the checkboxes of all columns that are to be included in the Data Source.

Add columns to create duplicates or to define different Data Types for one column. Calculated columns are not available in the Data Source Editor, import a data source to a knowledge set to use this functionality.

The columns FILE_TIME and FILE_NAME are included for CSV, Excel, XML type Data Sources

FILE_TIME Includes date and time of when the file was created or last modified
FILE_NAME Includes the full path to the files location on the server

 

Changes in data type will cause inorigo to interpret the columns as the given Data Type.

For inorigo to identify content as inorigo units (such as associations or association definitions), the column needs to hold the inorigo units unique IDs.

Click Add Columns to include the ticked columns in the Data Source.

 

The included columns appear in the Columns field as the image displayed below.

Names and Data Type can be modified.

Notice that these changes are saved in inorigo and will not be applied to the source. To view the columns as they are defined by the source, press Add Columns.

 

It’s possible to add manual columns below the list of available columns. Use the Source ID given in the right column to refer to an available column.

 

Filter Data Source

The query tool can be used to apply custom filters to the Data Source.

Press Define query to bring up the query tool – you can learn more about how to write queries here

Tick distinct rows to remove all duplicate rows.

 

Preview Data Source

Preview displays a preview of the first rows in the data source.

Use preview to verify that applied filters and changes to the columns works as intended.

Once finish is pressed, the Data Source is saved and will be available to use throughout the inorigo suite.

Save / Delete Connection or Data Source

If a Data Source/Connection has not yet been saved, name it and press Create. If changes are made to an existing Data Source/Connection, Save is available.

Saving a Data Source/Connection in the workbench always overwrites the previous save.

Create a Data Source/Connection
Apply changes (overwrites previous save)
Delete Data Source/Connection
Unable  to delete Data Source/Connection. Either because it has not yet been created, or because the Data Source/Connection is  in use. See Usages
Suggest Edit