Lỗi khi sử dụng onekey ghost

  -  

This article explores common troubleshooting methods for mapping data flows in Azure Data Factory.

Bạn đang xem: Lỗi khi sử dụng onekey ghost

General troubleshooting guidance

Check the status of your dataset connections. In each source and sink transformation, go khổng lồ the linked service for each dataset that you"re using & test the connections.Check the status of your file và table connections in the data flow designer. In debug mode, select Data Preview on your source transformations to ensure that you can access your data.If everything looks correct in data preview, go into the Pipeline designer & put your data flow in a Pipeline activity. Debug the pipeline for an end-to-end test.

Internal server errors

Specific scenarios that can cause internal server errors are shown as follows.

Scenario 1: Not choosing the appropriate compute size/type và other factors

Successful execution of data flows depends on many factors, including the compute size/type, numbers of source/sinks khổng lồ process, the partition specification, transformations involved, sizes of datasets, the data skewness và so on.

For more guidance, see Integration Runtime performance.

Scenario 2: Using debug sessions with parallel activities

When triggering a run using the data flow debug session with constructs like ForEach in the pipeline, multiple parallel runs can be submitted to lớn the same cluster. This situation can lead lớn cluster failure problems while running because of resource issues, such as being out of memory.

To submit a run with the appropriate integration runtime configuration defined in the pipeline activity after publishing the changes, select Trigger Now or Debug > Use Activity Runtime.

Scenario 3: Transient issues

Transient issues with microservices involved in the execution can cause the run to fail.

Configuring retries in the pipeline activity can resolve the problems caused by transient issues. For more guidance, see Activity Policy.

Common error codes & messages

This section lists common error codes and messages reported by mapping data flows in Azure Data Factory, along with their associated causes và recommendations.

Error code: DF-AdobeIntegration-InvalidMapToFilter

Message: Custom resource can only have one Key/Id mapped to filter.Cause: Invalid configurations are provided.Recommendation: In your AdobeIntegration settings, make sure that the custom resource can only have one Key/Id mapped lớn filter.

Error code: DF-AdobeIntegration-InvalidPartitionConfiguration

Message: Only single partition is supported. Partition schema may be RoundRobin or Hash.Cause: Invalid partition configurations are provided.Recommendation: In AdobeIntegration settings, confirm that only the single partition is set and partition schemas may be RoundRobin or Hash.

Error code: DF-AdobeIntegration-InvalidPartitionType

Message: Partition type has lớn be roundRobin.Cause: Invalid partition types are provided.Recommendation: Please update AdobeIntegration settings to lớn make your partition type is RoundRobin.

Error code: DF-AdobeIntegration-InvalidPrivacyRegulation

Message: Only privacy regulation that"s currently supported is "GDPR".Cause: Invalid privacy configurations are provided.Recommendation: Please update AdobeIntegration settings while only privacy "GDPR" is supported.

Error code: DF-AdobeIntegration-KeyColumnMissed

Message: Key must be specified for non-insertable operations.Cause: Key columns are missed.Recommendation: Update AdobeIntegration settings khổng lồ ensure key columns are specified for non-insertable operations.

Error code: DF-AzureDataExplorer-InvalidOperation

Message: Blob operation is not supported on older storage accounts. Creating a new storage account may fix the issue.Cause: Operation is not supported.Recommendation: Change Update method configuration as delete, update and upsert are not supported in Azure Data Explorer.

Error code: DF-AzureDataExplorer-ReadTimeout

Message: Operation timeout while reading data.Cause: Operation times out while reading data.Recommendation: Increase the value in Timeout option in source transformation settings.

Error code: DF-AzureDataExplorer-WriteTimeout

Message: Operation timeout while writing data.Cause: Operation times out while writing data.Recommendation: Increase the value in Timeout option in sink transformation settings.

Error code: DF-Blob-FunctionNotSupport

Message: This endpoint does not support BlobStorageEvents, SoftDelete or AutomaticSnapshot. Please disable these tài khoản features if you would like to use this endpoint.Cause: Azure Blob Storage events, soft delete or automatic snapshot is not supported in data flows if the Azure Blob Storage linked service is created with service principal or managed identity authentication.Recommendation: Disable Azure Blob Storage events, soft delete or automatic snapshot feature on the Azure Blob account, or use key authentication to lớn create the linked service.

Error code: DF-Blob-InvalidAccountConfiguration

Message: Either one of trương mục key or sas token should be specified.Cause: An invalid credential is provided in the Azure Blob linked service.Recommendation: Use either tài khoản key or SAS token for the Azure Blob linked service.

Error code: DF-Blob-InvalidAuthConfiguration

Message: Only one of the two auth methods (Key, SAS) can be specified.Cause: An invalid authentication method is provided in the linked service.Recommendation: Use key or SAS authentication for the Azure Blob linked service.

Error code: DF-Blob-InvalidCloudType

Message: Cloud type is invalid.Cause: An invalid cloud type is provided.Recommendation: Please check the cloud type in your related Azure Blob linked service.

Error code: DF-Cosmos-DeleteDataFailed

Message: Failed khổng lồ delete data from Azure Cosmos DB after 3 times retry.Cause: The throughput on the Azure Cosmos DB collection is small and leads khổng lồ meeting throttling or row data not existing in Azure Cosmos DB.Recommendation: Please take the following actions to lớn solve this problem:If the error is 404, make sure that the related row data exists in the Azure Cosmos DB collection.If the error is throttling, please increase the Azure Cosmos DB collection throughput or phối it lớn the automatic scale.If the error is request timed out, please phối "Batch size" in the Azure Cosmos DB sink to lớn smaller value, for example 1000.

Error code: DF-Cosmos-FailToResetThroughput

Message: Azure Cosmos DB throughput scale operation cannot be performed because another scale operation is in progress, please retry after sometime.Cause: The throughput scale operation of the Azure Cosmos DB can"t be performed because another scale operation is in progress.Recommendation: Log in to lớn Azure Cosmos DB account, and manually change container throughput to be auto scale or địa chỉ a custom activity after mapping data flows khổng lồ reset the throughput.

Error code: DF-Cosmos-IdPropertyMissed

Message: "id" property should be mapped for delete and update operations.Cause: The id property is missed for update & delete operations.Recommendation: Make sure that the input data has an id column in Azure Cosmos DB sink transformation settings. If not, use a select or derived column transformation lớn generate this column before the sink transformation.

Error code: DF-Cosmos-InvalidAccountConfiguration

Message: Either accountName or accountEndpoint should be specified.Cause: Invalid account information is provided.Recommendation: In the Azure Cosmos DB linked service, specify the tài khoản name or account endpoint.

Error code: DF-Cosmos-InvalidAccountKey

Message: The đầu vào authorization token can"t serve the request. Please kiểm tra that the expected payload is built as per the protocol, and check the key being used.Cause: There"s no enough permission to read/write Azure Cosmos DB data.Recommendation: Please use the read-write key to lớn access Azure Cosmos DB.

Error code: DF-Cosmos-InvalidConnectionMode

Message: Invalid connection mode.Cause: An invalid connection mode is provided.Recommendation: Confirm that the supported mode is Gateway and DirectHttps in Azure Cosmos DB settings.

Error code: DF-Cosmos-InvalidPartitionKey

Message: Partition key path cannot be empty for update & delete operations.Cause: The partition key path is empty for update and delete operations.Recommendation: Use the providing partition key in the Azure Cosmos DB sink settings.Message: Partition key is not mapped in sink for delete & update operations.Cause: An invalid partition key is provided.Recommendation: In Azure Cosmos DB sink settings, use the right partition key that is same as your container"s partition key.

Error code: DF-Cosmos-InvalidPartitionKeyContent

Message: partition key should start with /.Cause: An invalid partition key is provided.Recommendation: Ensure that the partition key start with / in Azure Cosmos DB sink settings, for example: /movieId.

Error code: DF-Cosmos-PartitionKeyMissed

Message: Partition key path should be specified for update và delete operations.Cause: The partition key path is missing in the Azure Cosmos DB sink.Recommendation: Provide the partition key in the Azure Cosmos DB sink settings.

Error code: DF-Cosmos-ResourceNotFound

Message: Resource not found.Cause: Invalid configuration is provided (for example, the partition key with invalid characters) or the resource doesn"t exist.

Error code: DF-Cosmos-ShortTypeNotSupport

Message: Short data type is not supported in Azure Cosmos DB.Cause: The short data type is not supported in the Azure Cosmos DB instance.Recommendation: add a derived column transformation khổng lồ convert related columns from short khổng lồ integer before using them in the Azure Cosmos DB sink transformation.

Error code: DF-CSVWriter-InvalidQuoteSetting

Message: Job failed while writing data with error: Quote character & escape character cannot be empty if column value contains column delimiterCause: Both quote characters & escape characters are empty when the column value contains column delimiter.Recommendation: set your quote character or escape character.

Error code: DF-Delimited-ColumnDelimiterMissed

Message: Column delimiter is required for parse.Cause: The column delimiter is missed.Recommendation: In your CSV settings, confirm that you have the column delimiter which is required for parse.

Error code: DF-Delimited-InvalidConfiguration

Message: Either one of empty lines or custom header should be specified.Cause: An invalid delimited configuration is provided.Recommendation: Please update the CSV settings khổng lồ specify one of empty lines or the custom header.

Error code: DF-DELTA-InvalidConfiguration

Message: Timestamp và version can"t be phối at the same time.Cause: The timestamp and version can"t be phối at the same time.Recommendation: phối the timestamp or version in the delta settings.

Error code: DF-Delta-InvalidProtocolVersion

Cause: Data flows don"t support this version of the Delta table protocol.Recommendation: Use a lower version of the Delta table protocol.

Error code: DF-DELTA-InvalidTableOperationSettings

Message: Recreate and truncate options can"t be both specified.Cause: Recreate and truncate options can"t be specified simultaneously.Recommendation: Update delta settings lớn have either recreate or truncate operation.

Error code: DF-DELTA-KeyColumnMissed

Message: Key column(s) should be specified for non-insertable operations.Cause: Key column(s) are missed for non-insertable operations.

Xem thêm: Unit 13 Festivals ( Speak Trang 123 Sgk Tiếng Anh Lớp 8, Speak Unit 13: Festivals

Recommendation: Specify key column(s) on delta sink to lớn have non-insertable operations.

Error code: DF-Dynamics-InvalidNullAlternateKeyColumn

Message: Any column value of alternate Key can"t be NULL.Cause: Your alternate key column value can"t be null.Recommendation: Confirm that your column value of your alternate key is not NULL.

Error code: DF-Dynamics-TooMuchAlternateKey

Cause: One lookup field with more than one alternate key reference is not valid.Recommendation: check your schema mapping & confirm that each lookup field has a single alternate key.

Error code: DF-Excel-DifferentSchemaNotSupport

Message: Read excel files with different schema is not supported now.Cause: Reading excel files with different schemas is not supported now.Recommendation: Please apply one of following options lớn solve this problem:Use ForEach + data flow activity to read Excel worksheets one by one.Update each worksheet schema to have the same columns manually before reading data.

Error code: DF-Excel-InvalidDataType

Message: Data type is not supported.Cause: The data type is not supported.Recommendation: Please change the data type to "string" for related input đầu vào data columns.

Error code: DF-Excel-InvalidFile

Message: Invalid excel file is provided while only .xlsx & .xls are supported.Cause: Invalid Excel files are provided.Recommendation: Use the wildcard to lớn filter, và get .xls và .xlsx Excel files before reading data.

Error code: DF-Excel-InvalidRange

Message: Invalid range is provided.Cause: An invalid range is provided.

Error code: DF-Excel-InvalidWorksheetConfiguration

Message: Excel sheet name and index cannot exist at the same time.Cause: The Excel sheet name và index are provided at the same time.Recommendation: check the parameter value và specify the sheet name or index to read the Excel data.

Error code: DF-Excel-WorksheetConfigMissed

Message: Excel sheet name or index is required.Cause: An invalid Excel worksheet configuration is provided.Recommendation: kiểm tra the parameter value và specify the sheet name or index khổng lồ read the Excel data.

Error code: DF-Excel-WorksheetNotExist

Message: Excel worksheet does not exist.Cause: An invalid worksheet name or index is provided.Recommendation: check the parameter value và specify a valid sheet name or index lớn read the Excel data.

Error code: DF-Executor-AcquireStorageMemoryFailed

Message: Transferring unroll memory to lớn storage memory failed. Cluster ran out of memory during execution. Please retry using an integration runtime with more cores and/or memory optimized compute type.Cause: The cluster has insufficient memory.Recommendation: Please use an integration runtime with more cores and/or the memory optimized compute type.

Error code: DF-Executor-BlockCountExceedsLimitError

Message: The uncommitted block count cannot exceed the maximum limit of 100,000 blocks. Check blob configuration.Cause: The maximum number of uncommitted blocks in a blob is 100,000.Recommendation: liên hệ the thuphikhongdung.vn product team for more details about this problem.

Error code: DF-Executor-BroadcastFailure

Message: Dataflow execution failed during broadcast exchange. Potential causes include misconfigured connections at sources or a broadcast join timeout error. Lớn ensure the sources are configured correctly, please kiểm tra the connection or run a source data preview in a Dataflow debug session. To avoid the broadcast join timeout, you can choose the "Off" broadcast option in the Join/Exists/Lookup transformations. If you intend to use the broadcast option to improve performance then make sure broadcast streams can produce data within 60 secs for debug runs & within 300 secs for job runs. If problem persists, tương tác customer support.

Cause:

The source connection/configuration error could lead to a broadcast failure in join/exists/lookup transformations.Broadcast has a mặc định timeout of 60 seconds in debug runs và 300 seconds in job runs. On the broadcast join, the stream chosen for the broadcast seems too large to lớn produce data within this limit. If a broadcast join is not used, the default broadcast done by a data flow can reach the same limit.

Recommendation:

Do data preview at sources to confirm the sources are well configured.Turn off the broadcast option or avoid broadcasting large data streams where the processing can take more than 60 seconds. Instead, choose a smaller stream khổng lồ broadcast.Large SQL/Data Warehouse tables & source files are typically bad candidates.In the absence of a broadcast join, use a larger cluster if the error occurs.If the problem persists, contact the customer support.

Error code: DF-Executor-BroadcastTimeout

Message: Broadcast join timeout error, make sure broadcast stream produces data within 60 secs in debug runs and 300 secs in job runs

Cause: Broadcast has a default timeout of 60 seconds on debug runs và 300 seconds on job runs. The stream chosen for broadcast is too large to produce data within this limit.

Recommendation: check the Optimize tab on your data flow transformations for join, exists, và lookup. The mặc định option for broadcast is Auto. If Auto is set, or if you"re manually setting the left or right side lớn broadcast under Fixed, you can either phối a larger Azure integration runtime (IR) configuration or turn off broadcast. For the best performance in data flows, we recommend that you allow Spark khổng lồ broadcast by using Auto và use a memory-optimized Azure IR.

If you"re running the data flow in a debug demo execution from a debug pipeline run, you might run into this condition more frequently. That"s because Azure Data Factory throttles the broadcast timeout to 60 seconds to lớn maintain a faster debugging experience. You can extend the timeout lớn the 300-second timeout of a triggered run. To vày so, you can use the Debug > Use Activity Runtime option lớn use the Azure IR defined in your Execute Data Flow pipeline activity.

Message: Broadcast join timeout error, you can choose "Off" of broadcast option in join/exists/lookup transformation to lớn avoid this issue. If you intend to lớn broadcast join option khổng lồ improve performance, then make sure broadcast stream can produce data within 60 secs in debug runs & 300 secs in job runs.

Cause: Broadcast has a mặc định timeout of 60 seconds in debug runs & 300 seconds in job runs. On the broadcast join, the stream chosen for broadcast is too large to lớn produce data within this limit. If a broadcast join isn"t used, the mặc định broadcast by dataflow can reach the same limit.

Recommendation: Turn off the broadcast option or avoid broadcasting large data streams for which the processing can take more than 60 seconds. Choose a smaller stream to broadcast. Large Azure SQL Data Warehouse tables & source files aren"t typically good choices. In the absence of a broadcast join, use a larger cluster if this error occurs.

Error code: DF-Executor-ColumnUnavailable

Message: Column name used in expression is unavailable or invalid.Cause: An invalid or unavailable column name is used in an expression.Recommendation: kiểm tra the column names used in expressions.

Error code: DF-Executor-Conversion

Message: Converting khổng lồ a date or time failed due khổng lồ an invalid characterCause: Data isn"t in the expected format.Recommendation: Use the correct data type.

Error code: DF-Executor-DriverError

Message: INT96 is legacy timestamp type, which is not supported by ADF Dataflow. Please consider upgrading the column type lớn the latest types.Cause: Driver error.Recommendation: INT96 is a legacy timestamp type that"s not supported by Azure Data Factory data flow. Consider upgrading the column type khổng lồ the latest type.

Error code: DF-Executor-FieldNotExist

Message: Field in struct does not exist.Cause: Invalid or unavailable field names are used in expressions.Recommendation: kiểm tra field names used in expressions.

Error code: DF-Executor-illegalArgument

Message: Please make sure that the access key in your Linked Service is correctCause: The trương mục name or access key is incorrect.Recommendation: Ensure that the tài khoản name or access key specified in your linked service is correct.

Error code: DF-Executor-IncorrectLinkedServiceConfiguration

Message: Possible causes are,The linked service is incorrectly configured as type "Azure Blob Storage" instead of "Azure DataLake Storage Gen2" và it has "Hierarchical namespace" enabled. Please create a new linked service of type "Azure DataLake Storage Gen2" for the storage account in question.Certain scenarios with any combinations of "Clear the folder", non-default "File name option", "Key" partitioning may fail with a Blob linked service on a "Hierarchical namespace" enabled storage account. You can disable these dataflow settings (if enabled) & try again in case you bởi vì not want lớn create a new Gen2 linked service.Cause: Delete operation on the Azure Data Lake Storage Gen2 trương mục failed since its linked service is incorrectly configured as Azure Blob Storage.Recommendation: Create a new Azure Data Lake Storage Gen2 linked service for the storage account. If that"s not feasible, some known scenarios lượt thích Clear the folder, non-default File name option, Key partitioning in any combinations may fail with an Azure Blob Storage linked service on a hierarchical namespace enabled storage account. You can disable these data flow settings if you enabled them và try again.

Error code: DF-Executor-InternalServerError

Message: Failed lớn execute dataflow with internal hệ thống error, please retry later. If issue persists, please contact thuphikhongdung.vn support for further assistanceCause: The data flow execution is failed because of the system error.

Error code: DF-Executor-InvalidColumn

Message: Column name needs khổng lồ be specified in the query, set an alias if using a SQL function.Cause: No column name is specified.Recommendation: mix an alias if you"re using a SQL function lượt thích min() or max().

Error code: DF-Executor-InvalidInputColumns

Message: The column in source configuration cannot be found in source data"s schema.Cause: Invalid columns are provided on the source.Recommendation: kiểm tra columns in the source configuration & make sure that it"s the subset of the source data"s schemas.

Error code: DF-Executor-InvalidOutputColumns

Message: The result has 0 đầu ra columns. Please ensure at least one column is mapped.Cause: No column is mapped.Recommendation: Please check the sink schema to ensure that at least one column is mapped.

Error code: DF-Executor-InvalidPartitionFileNames

Message: tệp tin names cannot have empty value(s) while tệp tin name option is set as per partition.Cause: Invalid partition tệp tin names are provided.Recommendation: Please check your sink settings to lớn have the right value of file names.

Error code: DF-Executor-InvalidPath

Message: Path does not resolve khổng lồ any file(s). Please make sure the file/folder exists & is not hidden.Cause: An invalid file/folder path is provided, which can"t be found or accessed.Recommendation: Please check the file/folder path, and make sure it is existed và can be accessed in your storage.

Error code: DF-Executor-InvalidStageConfiguration

Message: Storage with user assigned managed identity authentication in staging is not supported.Cause: An exception is happened because of invalid staging configuration.Recommendation: The user-assigned managed identity authentication is not supported in staging. Use a different authentication lớn create an Azure Data Lake Storage Gen2 or Azure Blob Storage linked service, then use it as staging in mapping data flows.

Error code: DF-Executor-InvalidType

Message: Please make sure that the type of parameter matches with type of value passed in. Passing float parameters from pipelines isn"t currently supported.Cause: Data types are incompatible between the declared type & the actual parameter value.Recommendation: kiểm tra that the parameter values passed into the data flow match the declared type.

Error code: DF-Executor-OutOfDiskSpaceError

Message: Internal hệ thống errorCause: The cluster is running out of disk space.Recommendation: Retry the pipeline. If doing so doesn"t resolve the problem, contact customer support.

Error code: DF-Executor-OutOfMemoryError

Message: Cluster ran into out of memory issue during execution, please retry using an integration runtime with bigger bộ vi xử lý core count and/or memory optimized compute typeCause: The cluster is running out of memory.

Error code: DF-Executor-OutOfMemorySparkBroadcastError

Message: Explicitly broadcasted dataset using left/right option should be small enough to lớn fit in node"s memory. You can choose broadcast option "Off" in join/exists/lookup transformation to lớn avoid this issue or use an integration runtime with higher memory.Cause: The kích cỡ of the broadcasted table far exceeds the limits of the node memory.Recommendation: The broadcast left/right option should be used only for smaller dataset kích thước which can fit into node"s memory, so make sure to lớn configure the node size appropriately or turn off the broadcast option.

Error code: DF-Executor-OutOfMemorySparkError

Message: The data may be too large to lớn fit in the memory.Cause: The form size of the data far exceeds the limit of the node memory.Recommendation: Increase the chip core count và switch to lớn the memory optimized compute type.

Error code: DF-Executor-ParseError

Message: Expression cannot be parsed.Cause: An expression generated parsing errors because of incorrect formatting.Recommendation: check the formatting in the expression.

Error code: DF-Executor-PartitionDirectoryError

Message: The specified source path has either multiple partitioned directories (for example, //a=10/b=20, //c=10/d=30) or partitioned directory with other file or non-partitioned directory (for example //a=10/b=20, /Directory 2/file1), remove partition root directory from source path and read it through separate source transformation.Cause: The source path has either multiple partitioned directories or a partitioned directory that has another tệp tin or non-partitioned directory.Recommendation: Remove the partitioned root directory from the source path and read it through separate source transformation.

Error code: DF-Executor-RemoteRPCClientDisassociated

Message: Job aborted due to lớn stage failure. Remote RPC client disassociated. Likely due lớn containers exceeding thresholds, or network issues.Cause: Data flow activity run failed because of transient network issues or one node in spark cluster ran out of memory.Recommendation: Use the following options to solve this problem:

Option-1: Use a powerful cluster (both drive and executor nodes have enough memory to lớn handle big data) to lớn run data flow pipelines with setting "Compute type" lớn "Memory optimized". The settings are shown in the picture below.

Xem thêm: 1 Kb Bằng Bao Nhiêu Byte, Bit, Mb, Gb Trong Đơn Vị Đo Lường Thông Tin

*

Option-3: Repartition your input đầu vào data. For the task running on the data flow spark cluster, one partition is one task và runs on one node. If data in one partition is too large, the related task running on the node needs khổng lồ consume more memory than the node itself, which causes failure. So you can use repartition lớn avoid data skew, and ensure that data size in each partition is average while the memory consumption isn"t too heavy.