Possible issue with --packages argument for submit-args

[andreimarinescu]

Hello everyone,

I'm trying to run a job using the following command:

dcos spark run --submit-args="--packages=org.apache.hadoop:hadoop-aws:2.7.1 https://external_domain/SparkPoc.py"

The task fails immediately with the following output from dcos spark status <job_id>:

Submission ID: driver-20161109150329-0013
Driver state: FINISHED
Last status: task_id {
  value: "driver-20161109150329-0013"
}
state: TASK_FAILED
message: "Failed to launch container: Failed to fetch all URIs for container \'12b2d97c-4499-4442-    8a54-33eb52ae0b2d\' with exit status: 256"
slave_id {
  value: "16766665-462f-4424-8796-6358785e8c59-S13"
}
timestamp: 1.4787038103792388E9
executor_id {
  value: "driver-20161109150329-0013"
}
source: SOURCE_SLAVE
reason: REASON_CONTAINER_LAUNCH_FAILED
uuid: "8\257\33575\aCV\204\353i\307G\2331"
container_status {
  network_infos {
    ip_addresses {
      ip_address: "10.0.1.132"
    }
  }
}

Looking at the mesos logs on that agent, I'm getting the following output. As far as I can see, all the jar dependencies are downloaded locally and they're only referenced from the mesos agent. Indeed, if I delete everything in /home/andrei/.ivy2, submitting the job takes substantially longer and the folder is then repopulated. Is this a bug, or am I doing anything wrong? I haven't found anything in the docs so far regarding the --packages parameter, running this on my local Spark cluster works.

1109 15:03:30.378451   582 fetcher.cpp:861] Begin fetcher log (stderr in sandbox) for container 12b2d97c-4499-4442-8a54-33eb52ae0b2d from running command: /opt/mesosphere/packages/mesos--19a545facb66e57dfe2bb905a001a58b7eaf6004/libexec/mesos/mesos-fetcher
I1109 15:03:30.262053  3921 logging.cpp:194] INFO level logging started!
I1109 15:03:30.274260  3921 fetcher.cpp:498] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/16766665-462f-4424-8796-6358785e8c59-S13","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.codehaus.jackson_jackson-jaxrs-1.9.13.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.sun.jersey_jersey-server-1.9.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.sun.xml.bind_jaxb-impl-2.2.3-1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.codehaus.jackson_jackson-core-asl-1.9.13.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.directory.api_api-util-1.0.0-M20.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.htrace_htrace-core-3.1.0-incubating.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.hadoop_hadoop-aws-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.google.code.findbugs_jsr305-3.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-collections_commons-collections-3.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.curator_curator-recipes-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.jcraft_jsch-0.1.42.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.hadoop_hadoop-auth-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.curator_curator-client-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/junit_junit-4.11.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.google.protobuf_protobuf-java-2.5.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.avro_avro-1.7.4.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.codehaus.jackson_jackson-xc-1.9.13.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-configuration_commons-configuration-1.6.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.hamcrest_hamcrest-core-1.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/javax.xml.bind_jaxb-api-2.2.2.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.slf4j_slf4j-api-1.7.10.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.commons_commons-math3-3.1.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.sun.jersey_jersey-json-1.9.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-httpclient_commons-httpclient-3.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-lang_commons-lang-2.6.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.amazonaws_aws-java-sdk-1.7.4.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.mortbay.jetty_jetty-util-6.1.26.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.google.code.gson_gson-2.2.4.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.httpcomponents_httpclient-4.2.5.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.mortbay.jetty_jetty-6.1.26.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-cli_commons-cli-1.2.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/javax.xml.stream_stax-api-1.0-2.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.slf4j_slf4j-log4j12-1.7.10.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/asm_asm-3.2.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/log4j_log4j-1.2.17.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-digester_commons-digester-1.8.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-beanutils_commons-beanutils-1.7.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/net.java.dev.jets3t_jets3t-0.9.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/javax.servlet_servlet-api-2.5.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.codehaus.jettison_jettison-1.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/joda-time_joda-time-2.9.5.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-net_commons-net-3.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-codec_commons-codec-1.4.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.zookeeper_zookeeper-3.4.6.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-beanutils_commons-beanutils-core-1.8.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.xerial.snappy_snappy-java-1.0.4.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.directory.api_api-asn1-api-1.0.0-M20.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/io.netty_netty-3.6.2.Final.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.curator_curator-framework-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.tukaani_xz-1.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.fasterxml.jackson.core_jackson-core-2.2.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/javax.servlet.jsp_jsp-api-2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.thoughtworks.paranamer_paranamer-2.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.hadoop_hadoop-common-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.httpcomponents_httpcore-4.2.5.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-io_commons-io-2.4.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.google.guava_guava-11.0.2.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/xmlenc_xmlenc-0.52.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"https:\/\/dl.dropboxusercontent.com\/u\/18356756\/SparkPoc.py"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.sun.jersey_jersey-core-1.9.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.commons_commons-compress-1.4.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.fasterxml.jackson.core_jackson-databind-2.2.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/javax.activation_activation-1.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.fasterxml.jackson.core_jackson-annotations-2.2.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/jline_jline-0.9.94.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/org.apache.hadoop_hadoop-annotations-2.7.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/com.jamesmurty.utils_java-xmlbuilder-0.4.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/home\/andrei\/.ivy2\/jars\/commons-logging_commons-logging-1.1.3.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/16766665-462f-4424-8796-6358785e8c59-S13\/frameworks\/f5a6d933-dab4-4ea8-a2f2-5615244948af-0001\/executors\/driver-20161109150329-0013\/runs\/12b2d97c-4499-4442-8a54-33eb52ae0b2d"}
I1109 15:03:30.277107  3921 fetcher.cpp:409] Fetching URI 'file:/home/andrei/.ivy2/jars/org.codehaus.jackson_jackson-jaxrs-1.9.13.jar'
I1109 15:03:30.277133  3921 fetcher.cpp:250] Fetching directly into the sandbox directory
I1109 15:03:30.277163  3921 fetcher.cpp:187] Fetching URI 'file:/home/andrei/.ivy2/jars/org.codehaus.jackson_jackson-jaxrs-1.9.13.jar'
Failed to fetch 'file:/home/andrei/.ivy2/jars/org.codehaus.jackson_jackson-jaxrs-1.9.13.jar': A relative path was passed for the resource but the Mesos framework home was not specified. Please either provide this config option or avoid using a relative path

[gabidobo]

Using the --packages argument seems to actually populate the --jars argument with local paths to all .jar dependencies, and that won't work inside the cluster.

What I ended up doing is a combination of the spark.mesos.uris and spark.driver.extraClassPath settings - the former to download the dependencies inside the sandbox and the latter to point the driver classpath to them. Something like:

dcos spark run --submit-args='--conf spark.mesos.uris=http://central.maven.org/maven2/org/apache/spark/spark-streaming-kinesis-asl-assembly_2.10/2.0.0/spark-streaming-kinesis-asl-assembly_2.10-2.0.0.jar,http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.3/hadoop-aws-2.7.3.jar --conf spark.driver.extraClassPath=/mnt/mesos/sandbox/spark-streaming-kinesis-asl-assembly_2.10-2.0.0.jar:/mnt/mesos/sandbox/hadoop-aws-2.7.3.jar  http://domain/SparkPoc.py 20' --verbose

[bogdangi]

Hi, I had the same issue with packages.
In my case I was not able to use mongodb-spark-connector package.
On my local environments I just run command like this

spark-submit \
        --packages org.mongodb.spark:mongo-spark-connector_2.10:2.0.0 \
        count_something.py

But to be able run it with DC/OS Spark I have to do the follow:

dcos spark run \
        --submit-args=""`
`"--conf spark.mesos.uris=http://repo1.maven.org/maven2/org/mongodb/spark/mongo-spark-connector_2.10/2.0.0/mongo-spark-connector_2.10-2.0.0.jar,http://repo1.maven.org/maven2/
    org/mongodb/mongo-java-driver/3.2.2/mongo-java-driver-3.2.2.jar "`
`"--conf spark.driver.extraClassPath=/mnt/mesos/sandbox/mongo-spark-connector_2.10-2.0.0.jar:/mnt/mesos/sandbox/mongo-java-driver-3.2.2.jar "`
`"--conf spark.executor.extraClassPath=/mnt/mesos/sandbox/mongo-spark-connector_2.10-2.0.0.jar:/mnt/mesos/sandbox/mongo-java-driver-3.2.2.jar "`
`"https://example.com/spark/do_something.py" \
        --verbose

As you can see it cannot resolve dependicies and this why I had to provide also jar for mongo-java-driver and also I had to set in additional spark.executor.extraClassPath

I think it is only because dcos spark bypasses --packages as download resource uri.

[Bumangues]

@bogdangi
Did you have to build the mongo-spark connector throughout your cluster?

[skonto]

@andreimarinescu there is a fix here for driver dependencies: apache/spark#18587. It will go in 2.3. Also planning to update the cli, --packages is currently unsupported along with some other flags.