The following section describes data extraction to a Huawei Cloud Object Storage Service (OBS).

Warning! File Fragments in the Cloud Storage
Huawei Cloud OBS destination uses multipart upload. That means that data is uploaded in fragments that are merged into a single file at the end of the extraction. When an extraction fails due to connection issues, the request to cancel the multipart upload can fail. In that case the uploaded fragments must be deleted manually, see Huawei Cloud Support: Deleting Fragments Directly.

Connection #

Adding a Destination #

  1. In the main window of the Designer, navigate to Server > Manage Destinations. The window “Manage Destinations” opens.
  2. Click [Add] to create a new destination. The window “Destination Details” opens.
  3. Enter a Name for the destination.
  4. Select the destination Type from the drop-down menu.

Destination Details #

huawei-destination-details

Authentication #

Access Key ID (AK)
Enter the access key of the Huawei Cloud OBS account. For more information on how to create access keys, see Huawei Cloud Support: Creating Access Keys (AK and SK)

Secret Access Key ID (SK)
Enter the secret access key of the Huawei Cloud OBS account. For more information on how to create access keys, see Huawei Cloud Support: Creating Access Keys (AK and SK)

Region
Select the region of the data storage.

Connect
Click [Connect] to establish a connection to the storage account. If the connection is successful, “Connected” is displayed next to the button.

Bucket #

Bucket
This setting only becomes available after a connection to the storage account is established.
Select a bucket. The SAP data is extracted into the selected bucket. Click refresh to refresh the list of available buckets.

Misc #

Folder path
Option to create a folder structure within the container for saving files, see also Destination Settings > Folder Path.
For creating a single folder, enter a folder name without slashes: [folder]
Subfolders are supported and can be defined using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

Using Script Expressions as Dynamic Folder Paths

Script expressions can be used to generate a dynamic folder path. This allows generating folder paths that are composed of an extraction’s properties, e.g., extraction name, SAP source object. The described scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

Input Description
#{Source.Name}# Name of the extraction’s SAP source.
#{Extraction.ExtractionName}# Name of the extraction.
#{Extraction.Type}# Extraction type (Table, ODP, BAPI, etc.).
#{Extraction.SapObjectName}# Name of the SAP object the extraction is extracting data from.
#{Extraction.Timestamp}# Timestamp of the extraction.
#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}# Removes the first slash ‘/’ of an SAP object.
Example: /BIO/TMATERIAL to BIO/TMATERIAL - prevents creating an empty folder in a file path.
#{Extraction.SapObjectName.Replace('/', '_')}# Replaces all slashes ‘/’ of an SAP object.
Example: /BIO/TMATERIAL to _BIO_TMATERIAL - prevents splitting the SAP object name by folders in a file path.
#{Extraction.Context}# Only for ODP extractions: returns the context of the ODP object (SAPI, ABAP_CDS, etc).
#{Extraction.Fields["[NameSelectionFields]"].Selections[0].Value}# Only for ODP extractions: returns the input value of a defined selection / filter.
#{Odp.UpdateMode}# Only for ODP extractions: returns the update mode (Delta, Full, Repeat) of the extraction.
#{TableExtraction.WhereClause}# Only for Table extractions: returns the WHERE clause of the extraction.
#{Extraction.Fields["[0D_NW_CODE]"].Selections[0].Value}# Only for BWCube extractions (MDX mode): returns the input value of a defined selection.
#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].LowerValue}# Only for BWCube extractions (MDX mode): returns the lower input value of a defined selection range.
#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].UpperValue}# Only for BWCube extractions (MDX mode): returns the upper input value of a defined selection range.
#{Extraction.Fields["0D_NW_CODE"].Selections[0].Value}# Only for BWCube extractions (BICS mode): returns the input value of a defined selection.
#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].LowerValue}# Only for BWCube extractions (BICS mode): returns the lower input value of a defined selection range.
#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].UpperValue}# Only for BWCube extractions (BICS mode): returns the upper input value of a defined selection range.

File Format #

File type
Select the required file format. Possible options are: CSV, Parquet and JSON.

huawei-destination-details2

CVS Settings

The settings for file type CSV correspond to the Flat File CSV settings.

Parquet Settings

The following compatibility modes are available:

  • Pure
  • Spark
  • BigQuery

Spark does not support the data types used in pure mode, so other data types need to be used. Special characters (e.g. ~) can be used in column names, when the option Allow special characters in column name is activated.

huawei-destination-details3

SAP Pure / BigQuery Spark
INT1 UINT_8 INT16
TIMS TIME_MILLIS UTF8

Connection Retry and Rollback #

Connection retry and rollback are built-in functions of the Huawei Cloud OBS destination. They are activated by default.

Connection retry is a functionality that prevents extractions from failing if the connection to Huawei is interrupted.

If an exception is thrown, Xtract Universal uses an exponential retry strategy to reestablish connection to Huawei. The selected exponential retry strategy results in 7 retry attempts and an overall timespan of 140 seconds. If a connection is not established during this timespan, the extraction fails.

Rollback covers scenarios where extractions do not fail due to connection failures to Huawei but e.g. due to an error when connecting to SAP. In those cases Xtract Universal tries to remove any files from the Huawei Cloud storage that were created in the course of the extraction.

Settings #

Opening the Destination Settings #

  1. Create or select an existing extraction, see Getting Started with Xtract Universal.
  2. Click [Destination]. The window “Destination Settings” opens. Destination-settings

The following settings can be defined for the destination:

Destination Settings #

huawei-destination-settings

File Name #

File Name determines the name of the target table. You have the following options:

  • Same as name of SAP object: Copy the name of the SAP object
  • Same as name of extraction: Adopt name of extraction
  • Custom: Define a name of your choice
  • Append timestamp: Add the timestamp in the UTC format (_YYYY_MM_DD_hh_mm_ss_fff) to the file name of the extraction

Note: If the file name contains the character /, it is replaced with an underscore.

Using Script Expressions as Dynamic File Names

Script expressions can be used to generate a dynamic file name. This allows generating file names that are composed of an extraction’s properties, e.g. extraction name, SAP source object. This scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

Input Description
#{Source.Name}# Name of the extraction’s SAP source.
#{Extraction.ExtractionName}# Name of the extraction.
#{Extraction.Type}# Extraction type (Table, ODP, BAPI, etc.).
#{Extraction.SapObjectName}# Name of the SAP object the extraction is extracting data from.
#{Extraction.Timestamp}# Timestamp of the extraction.
#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}# Removes the first slash ‘/’ of an SAP object.
Example: /BIO/TMATERIAL to BIO/TMATERIAL - prevents creating an empty folder in a file path.
#{Extraction.SapObjectName.Replace('/', '_')}# Replaces all slashes ‘/’ of an SAP object.
Example /BIO/TMATERIAL to _BIO_TMATERIAL - prevents splitting the SAP object name by folders in a file path.
#{Extraction.Context}# Only for ODP extractions: returns the context of the ODP object (SAPI, ABAP_CDS, etc).
#{Extraction.Fields["[NameSelectionFiels]"].Selections[0].Value}# Only for ODP extractions: returns the input value of a defined selection / filter.
#{Odp.UpdateMode}# Only for ODP extractions: returns the update mode (Delta, Full, Repeat) of the extraction.
#{TableExtraction.WhereClause}# Only for Table extractions: returns the WHERE clause of the extraction.

Column Name Style #

Defines the style of the column name. Following options are available:

column_name_style_options

  • Code: The SAP technical column name is used as column name in the destination e.g., MAKTX.
  • PrefixedCode: The SAP technical column name is prefixed by SAP object name and the tilde character e.g., MAKT~MAKTX
  • CodeAndText: The SAP technical column name and the SAP description separated by an underscore are used as column name in the destination e.g., MAKTX_Material Description (Short Text).
  • TextAndCode: The SAP description and the SAP technical column name description separated by an underscore are used as column name in the destination e.g., Material Description (Short Text)_MAKTX.
  • Text: The SAP description is used as column name in the destination e.g., Material Description (Short Text).

Date conversion #

Convert date strings
Converts the character-type SAP date (YYYYMMDD, e.g., 19900101) to a special date format (YYYY-MM-DD, e.g., 1990-01-01). Target data uses a real date data-type and not the string data-type to store dates.

Convert invalid dates to
If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value.

When converting the SAP date the two special cases 00000000 and 9999XXXX are checked at first.

Convert 00000000 to
Converts the SAP date 00000000 to the entered value.

Convert 9999XXXX to
Converts the SAP date 9999XXXX to the entered value.

Folder #

Option to create a folder structure within the container for saving files, see also Connection > Folder Path.
For creating a single folder, enter a folder name without slashes: [folder]
Subfolders are supported and can be defined using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

Using Script Expressions as Dynamic Folder Paths

Script expressions can be used to generate a dynamic folder path. This allows generating folder paths that are composed of an extraction’s properties, e.g., extraction name, SAP source object. The described scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

Input Description
#{Source.Name}# Name of the extraction’s SAP source.
#{Extraction.ExtractionName}# Name of the extraction.
#{Extraction.Type}# Extraction type (Table, ODP, BAPI, etc.).
#{Extraction.SapObjectName}# Name of the SAP object the extraction is extracting data from.
#{Extraction.Timestamp}# Timestamp of the extraction.
#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}# Removes the first slash ‘/’ of an SAP object.
Example: /BIO/TMATERIAL to BIO/TMATERIAL - prevents creating an empty folder in a file path.
#{Extraction.SapObjectName.Replace('/', '_')}# Replaces all slashes ‘/’ of an SAP object.
Example: /BIO/TMATERIAL to _BIO_TMATERIAL - prevents splitting the SAP object name by folders in a file path.
#{Extraction.Context}# Only for ODP extractions: returns the context of the ODP object (SAPI, ABAP_CDS, etc).
#{Extraction.Fields["[NameSelectionFields]"].Selections[0].Value}# Only for ODP extractions: returns the input value of a defined selection / filter.
#{Odp.UpdateMode}# Only for ODP extractions: returns the update mode (Delta, Full, Repeat) of the extraction.
#{TableExtraction.WhereClause}# Only for Table extractions: returns the WHERE clause of the extraction.
#{Extraction.Fields["[0D_NW_CODE]"].Selections[0].Value}# Only for BWCube extractions (MDX mode): returns the input value of a defined selection.
#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].LowerValue}# Only for BWCube extractions (MDX mode): returns the lower input value of a defined selection range.
#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].UpperValue}# Only for BWCube extractions (MDX mode): returns the upper input value of a defined selection range.
#{Extraction.Fields["0D_NW_CODE"].Selections[0].Value}# Only for BWCube extractions (BICS mode): returns the input value of a defined selection.
#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].LowerValue}# Only for BWCube extractions (BICS mode): returns the lower input value of a defined selection range.
#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].UpperValue}# Only for BWCube extractions (BICS mode): returns the upper input value of a defined selection range.

Compression #

None
The data is transferred uncompressed and stored as a csv file.

gzip
The data is transferred compressed and stored as a gz file.

Note: Compression is only available for the csv file format, see Connection: File Format.

File Splitting #

File Splitting
Writes extraction data of a single extraction to multiple files in the cloud. Each filename is appended by _part[nnn].

Max. file size
The value set in Max. file size determines the maximum size of each stored file.

Note: The option Max. file size does not apply to gzip files. The size of a gzipped file cannot be determined in advance.