Skip to content

Latest commit

 

History

History
35 lines (25 loc) · 1.47 KB

solution-sample.md

File metadata and controls

35 lines (25 loc) · 1.47 KB
title description services author ms.author ms.service ms.custom ms.topic ms.date
Azure HDInsight Solutions | Apache Spark | IllegalArgumentException
Learn how to resolve an IllegalArgumentException when running Apache Spark jobs using Azure Data Factory.
hdinsight
confusionblinds
sunilkc
hdinsight
troubleshooting
conceptual
10/30/2018

Azure HDInsight Solutions | Spark | IllegalArgumentException

Scenario: Spark activity running in Azure Data Factory fails with IllegalArgumentException

Issue

You receive the following exception when trying to execute a Spark activity in an Azure Data Factory pipeline:

Exception in thread "main" java.lang.IllegalArgumentException: 
Wrong FS: wasbs://[email protected]/spark-examples_2.11-2.1.0.jar, expected: wasbs://[email protected]

Cause

A Spark job will fail if the application jar file is not located in the Spark cluster’s default/primary storage.

This is a known issue with the Spark open source framework tracked in this bug: Spark job fails if fs.defaultFS and application jar are different url

This issue has been resolved in Spark 2.3.0

Solution

Make sure the application jar is stored on the default/primary storage for the HDInsight cluster. In case of Azure Data Factory make sure the ADF linked service is pointed to the HDInsight default container rather than a secondary container.