2 votes

rakeeee avatar

Spark-submit dans EMR ClassNotFoundException java

Demandé el 18 de Avril, 2019
Quand la question a-t-elle été
771 affichage
Nombre de visites la question a
1 Réponses
Nombre de réponses aux questions
Résolu
Situation réelle de la question

J'ai créé un bundled jar à partir d'une application spring boot dans laquelle je crée une session spark pour effectuer certaines opérations.

Ensuite, j'expédie ce jar vers AWS EMR et je l'exécute en tant que spark-submit o java -jar exécutable qui ne fonctionne pas.

Voici la structure de mon bocal lorsque j'utilise jar -tf <jar name>

META-INF/
META-INF/MANIFEST.MF
org/
org/springframework/
org/springframework/boot/
org/springframework/boot/loader/
org/springframework/boot/loader/data/
org/springframework/boot/loader/data/RandomAccessDataFile$DataInputStream.class
org/springframework/boot/loader/jar/
org/springframework/boot/loader/jar/CentralDirectoryEndRecord.class
org/springframework/boot/loader/data/RandomAccessDataFile$FileAccess.class
org/springframework/boot/loader/jar/CentralDirectoryParser.class
org/springframework/boot/loader/data/RandomAccessDataFile$1.class
org/springframework/boot/loader/jar/ZipInflaterInputStream.class
org/springframework/boot/loader/data/RandomAccessDataFile.class
org/springframework/boot/loader/jar/Handler.class
org/springframework/boot/loader/data/RandomAccessData.class
org/springframework/boot/loader/archive/
org/springframework/boot/loader/archive/Archive$EntryFilter.class
org/springframework/boot/loader/jar/Bytes.class
org/springframework/boot/loader/archive/Archive.class
org/springframework/boot/loader/archive/JarFileArchive.class
org/springframework/boot/loader/PropertiesLauncher$PrefixMatchingArchiveFilter.class
org/springframework/boot/loader/PropertiesLauncher$ArchiveEntryFilter.class
org/springframework/boot/loader/PropertiesLauncher$1.class
org/springframework/boot/loader/PropertiesLauncher.class
org/springframework/boot/loader/Launcher.class
org/springframework/boot/loader/MainMethodRunner.class
org/springframework/boot/loader/ExecutableArchiveLauncher.class
org/springframework/boot/loader/WarLauncher.class
org/springframework/boot/loader/JarLauncher.class
org/springframework/boot/loader/util/
org/springframework/boot/loader/util/SystemPropertyUtils.class
org/springframework/boot/loader/LaunchedURLClassLoader$UseFastConnectionExceptionsEnumeration.class
org/springframework/boot/loader/LaunchedURLClassLoader.class
org/springframework/boot/loader/jar/JarURLConnection$1.class
org/springframework/boot/loader/archive/JarFileArchive$EntryIterator.class
org/springframework/boot/loader/jar/JarURLConnection$JarEntryName.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator$EntryComparator.class
org/springframework/boot/loader/jar/JarURLConnection.class
org/springframework/boot/loader/jar/JarFile$1.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntry.class
org/springframework/boot/loader/archive/ExplodedArchive$1.class
org/springframework/boot/loader/jar/JarFile$2.class
org/springframework/boot/loader/jar/JarFile$JarFileType.class
org/springframework/boot/loader/archive/ExplodedArchive.class
org/springframework/boot/loader/archive/JarFileArchive$JarFileEntry.class
org/springframework/boot/loader/jar/JarFile.class
org/springframework/boot/loader/jar/JarEntry.class
org/springframework/boot/loader/jar/FileHeader.class
org/springframework/boot/loader/jar/StringSequence.class
org/springframework/boot/loader/jar/AsciiBytes.class
org/springframework/boot/loader/jar/JarFileEntries$1.class
org/springframework/boot/loader/jar/JarFileEntries$EntryIterator.class
org/springframework/boot/loader/jar/JarFileEntries.class
org/springframework/boot/loader/jar/CentralDirectoryVisitor.class
org/springframework/boot/loader/jar/JarEntryFilter.class
org/springframework/boot/loader/jar/CentralDirectoryFileHeader.class
org/springframework/boot/loader/archive/Archive$Entry.class
BOOT-INF/
BOOT-INF/classes/
BOOT-INF/classes/com/
BOOT-INF/classes/com/ati/
BOOT-INF/classes/com/ati/media/
BOOT-INF/classes/com/ati/media/ref/
BOOT-INF/classes/com/ati/media/ref/scd/
BOOT-INF/classes/com/ati/media/ref/scd/config/
BOOT-INF/classes/com/ati/media/ref/scd/controllers/
BOOT-INF/classes/com/ati/media/ref/scd/error/
BOOT-INF/classes/com/ati/media/ref/scd/repositories/
BOOT-INF/classes/com/ati/media/ref/scd/services/
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/
META-INF/maven/
META-INF/maven/com.ati.media.ref/
META-INF/maven/com.ati.media.ref/ref-svc-scd/
BOOT-INF/classes/application-local.yml
BOOT-INF/classes/bootstrap.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SwaggerConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleExceptionHandler.class
BOOT-INF/classes/com/ati/media/ref/scd/ScheduleApplication.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleServiceImpl.class
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.xml
BOOT-INF/classes/bootstrap-docker.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/ScheduleConstants.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleCreationException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.xml
BOOT-INF/classes/com/ati/media/ref/scd/services/ConfigurationService.class
BOOT-INF/classes/git.properties
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.properties
BOOT-INF/classes/application.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SparkConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleEditException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/SparkAthenaRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleService.class
META-INF/build-info.properties
BOOT-INF/classes/bootstrap-local.yml
BOOT-INF/classes/com/ati/media/ref/scd/controllers/ScheduleController.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/AthenaQueryBuilder.class
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/ScheduleValidator.class
META-INF/build.properties
BOOT-INF/lib/
BOOT-INF/lib/core-tech-starter-web-base-1.0.0.jar
BOOT-INF/lib/spring-boot-starter-web-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-json-2.0.3.RELEASE.jar
BOOT-INF/lib/jackson-datatype-jdk8-2.9.6.jar
BOOT-INF/lib/jackson-datatype-jsr310-2.9.6.jar
BOOT-INF/lib/jackson-module-parameter-names-2.9.6.jar
BOOT-INF/lib/spring-boot-starter-tomcat-2.0.3.RELEASE.jar
BOOT-INF/lib/tomcat-embed-core-8.5.31.jar
BOOT-INF/lib/tomcat-embed-el-8.5.31.jar
BOOT-INF/lib/tomcat-embed-websocket-8.5.31.jar
BOOT-INF/lib/hibernate-validator-6.0.10.Final.jar
BOOT-INF/lib/spring-web-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-webmvc-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-expression-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-webflux-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-reactor-netty-2.0.3.RELEASE.jar
BOOT-INF/lib/reactor-netty-0.7.8.RELEASE.jar
BOOT-INF/lib/netty-codec-http-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-4.1.25.Final.jar
BOOT-INF/lib/netty-buffer-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-4.1.25.Final.jar
BOOT-INF/lib/netty-resolver-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-proxy-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-socks-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar
BOOT-INF/lib/netty-common-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-unix-common-4.1.25.Final.jar
BOOT-INF/lib/spring-webflux-5.0.7.RELEASE.jar
BOOT-INF/lib/reactor-core-3.1.8.RELEASE.jar
BOOT-INF/lib/nio-multipart-parser-1.1.0.jar
BOOT-INF/lib/nio-stream-storage-1.1.3.jar
BOOT-INF/lib/spring-boot-starter-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/micrometer-core-1.0.5.jar
BOOT-INF/lib/HdrHistogram-2.1.10.jar
BOOT-INF/lib/LatencyUtils-2.0.3.jar
BOOT-INF/lib/spring-boot-starter-log4j2-2.0.3.RELEASE.jar
BOOT-INF/lib/log4j-slf4j-impl-2.10.0.jar
BOOT-INF/lib/log4j-core-2.10.0.jar
BOOT-INF/lib/log4j-jul-2.10.0.jar
BOOT-INF/lib/log4j-1.2-api-2.7.jar
BOOT-INF/lib/log4j-api-2.10.0.jar
BOOT-INF/lib/lombok-1.18.0.jar
BOOT-INF/lib/micrometer-registry-prometheus-1.0.3.jar
BOOT-INF/lib/simpleclient_common-0.3.0.jar
BOOT-INF/lib/simpleclient-0.3.0.jar
BOOT-INF/lib/spring-cloud-starter-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-sleuth-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-sleuth-core-2.0.0.RELEASE.jar
BOOT-INF/lib/aspectjrt-1.8.13.jar
BOOT-INF/lib/brave-5.1.0.jar
BOOT-INF/lib/brave-context-log4j2-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-web-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-http-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-rabbit-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-kafka-clients-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpasyncclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-webmvc-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-servlet-5.1.0.jar
BOOT-INF/lib/spring-cloud-sleuth-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/zipkin-2.9.3.jar
BOOT-INF/lib/zipkin-reporter-2.7.3.jar
BOOT-INF/lib/zipkin-sender-kafka11-2.7.3.jar
BOOT-INF/lib/zipkin-sender-amqp-client-2.7.3.jar
BOOT-INF/lib/core-tech-starter-amqp-1.0.0.jar
BOOT-INF/lib/spring-cloud-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-validation-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-messaging-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-integration-core-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-integration-jmx-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-amqp-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-rabbit-2.0.4.RELEASE.jar
BOOT-INF/lib/spring-amqp-2.0.4.RELEASE.jar
BOOT-INF/lib/amqp-client-5.1.2.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-amqp-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-config-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-context-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-rsa-1.0.5.RELEASE.jar
BOOT-INF/lib/bcpkix-jdk15on-1.56.jar
BOOT-INF/lib/spring-cloud-config-client-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-aop-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-aop-5.0.7.RELEASE.jar
BOOT-INF/lib/aspectjweaver-1.8.13.jar
BOOT-INF/lib/core-tech-lib-core-1.0.0.jar
BOOT-INF/lib/spring-retry-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-openfeign-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-openfeign-core-2.0.0.RELEASE.jar
BOOT-INF/lib/feign-core-9.5.1.jar
BOOT-INF/lib/feign-slf4j-9.5.1.jar
BOOT-INF/lib/feign-hystrix-9.5.1.jar
BOOT-INF/lib/feign-java8-9.5.1.jar
BOOT-INF/lib/spring-cloud-starter-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/ribbon-2.2.5.jar
BOOT-INF/lib/ribbon-transport-2.2.5.jar
BOOT-INF/lib/rxnetty-contexts-0.4.9.jar
BOOT-INF/lib/rxnetty-servo-0.4.9.jar
BOOT-INF/lib/rxnetty-0.4.9.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final.jar
BOOT-INF/lib/ribbon-core-2.2.5.jar
BOOT-INF/lib/ribbon-httpclient-2.2.5.jar
BOOT-INF/lib/jersey-client-1.19.1.jar
BOOT-INF/lib/jersey-apache-client4-1.19.1.jar
BOOT-INF/lib/servo-core-0.12.21.jar
BOOT-INF/lib/netflix-commons-util-0.3.0.jar
BOOT-INF/lib/ribbon-loadbalancer-2.2.5.jar
BOOT-INF/lib/netflix-statistics-0.1.1.jar
BOOT-INF/lib/commons-text-1.3.jar
BOOT-INF/lib/commons-collections4-4.1.jar
BOOT-INF/lib/core-tech-starter-web-api-1.0.0.jar
BOOT-INF/lib/springfox-swagger2-2.9.2.jar
BOOT-INF/lib/swagger-annotations-1.5.20.jar
BOOT-INF/lib/swagger-models-1.5.20.jar
BOOT-INF/lib/springfox-spi-2.9.2.jar
BOOT-INF/lib/springfox-core-2.9.2.jar
BOOT-INF/lib/springfox-schema-2.9.2.jar
BOOT-INF/lib/springfox-swagger-common-2.9.2.jar
BOOT-INF/lib/springfox-spring-web-2.9.2.jar
BOOT-INF/lib/guava-20.0.jar
BOOT-INF/lib/classmate-1.3.4.jar
BOOT-INF/lib/spring-plugin-core-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-plugin-metadata-1.2.0.RELEASE.jar
BOOT-INF/lib/mapstruct-1.2.0.Final.jar
BOOT-INF/lib/springfox-swagger-ui-2.9.2.jar
BOOT-INF/lib/spring-cloud-starter-netflix-hystrix-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/archaius-core-0.7.6.jar
BOOT-INF/lib/hystrix-core-1.5.12.jar
BOOT-INF/lib/rxjava-1.3.8.jar
BOOT-INF/lib/hystrix-serialization-1.5.12.jar
BOOT-INF/lib/jackson-module-afterburner-2.9.6.jar
BOOT-INF/lib/hystrix-metrics-event-stream-1.5.12.jar
BOOT-INF/lib/hystrix-javanica-1.5.12.jar
BOOT-INF/lib/asm-5.0.4.jar
BOOT-INF/lib/rxjava-reactive-streams-1.2.1.jar
BOOT-INF/lib/reactive-streams-1.0.2.jar
BOOT-INF/lib/spring-cloud-netflix-hystrix-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-commons-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-crypto-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-stream-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/postgresql-42.2.2.jar
BOOT-INF/lib/athenaJDBC-1.1.0.jar
BOOT-INF/lib/lib-client-scd-0.0.0.jar
BOOT-INF/lib/core-tech-starter-web-client-1.0.0.jar
BOOT-INF/lib/hibernate-jpa-2.1-api-1.0.2.Final.jar
BOOT-INF/lib/gson-2.8.5.jar
BOOT-INF/lib/spring-boot-starter-data-jpa-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/javax.annotation-api-1.3.2.jar
BOOT-INF/lib/snakeyaml-1.19.jar
BOOT-INF/lib/spring-boot-starter-jdbc-2.0.3.RELEASE.jar
BOOT-INF/lib/HikariCP-2.7.9.jar
BOOT-INF/lib/spring-jdbc-5.0.7.RELEASE.jar
BOOT-INF/lib/hibernate-core-5.2.17.Final.jar
BOOT-INF/lib/jboss-logging-3.3.2.Final.jar
BOOT-INF/lib/javassist-3.22.0-GA.jar
BOOT-INF/lib/antlr-2.7.7.jar
BOOT-INF/lib/jandex-2.0.3.Final.jar
BOOT-INF/lib/dom4j-1.6.1.jar
BOOT-INF/lib/hibernate-commons-annotations-5.0.1.Final.jar
BOOT-INF/lib/javax.transaction-api-1.2.jar
BOOT-INF/lib/spring-data-jpa-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-data-commons-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-orm-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-context-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-beans-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-aspects-5.0.7.RELEASE.jar
BOOT-INF/lib/mybatis-spring-boot-starter-1.3.1.jar
BOOT-INF/lib/mybatis-spring-boot-autoconfigure-1.3.1.jar
BOOT-INF/lib/mybatis-3.4.5.jar
BOOT-INF/lib/mybatis-spring-1.3.1.jar
BOOT-INF/lib/spring-boot-starter-jooq-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-tx-5.0.7.RELEASE.jar
BOOT-INF/lib/jooq-3.10.7.jar
BOOT-INF/lib/byte-buddy-1.7.11.jar
BOOT-INF/lib/objenesis-2.6.jar
BOOT-INF/lib/spring-core-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-jcl-5.0.7.RELEASE.jar
BOOT-INF/lib/core-tech-starter-web-request-details-1.0.0.jar
BOOT-INF/lib/spark-core_2.11-2.3.0.jar
BOOT-INF/lib/avro-1.7.7.jar
BOOT-INF/lib/jackson-core-asl-1.9.13.jar
BOOT-INF/lib/paranamer-2.3.jar
BOOT-INF/lib/commons-compress-1.4.1.jar
BOOT-INF/lib/xz-1.0.jar
BOOT-INF/lib/avro-mapred-1.7.7-hadoop2.jar
BOOT-INF/lib/avro-ipc-1.7.7.jar
BOOT-INF/lib/avro-ipc-1.7.7-tests.jar
BOOT-INF/lib/chill_2.11-0.8.4.jar
BOOT-INF/lib/kryo-shaded-3.0.3.jar
BOOT-INF/lib/minlog-1.3.0.jar
BOOT-INF/lib/chill-java-0.8.4.jar
BOOT-INF/lib/xbean-asm5-shaded-4.4.jar
BOOT-INF/lib/hadoop-client-2.6.5.jar
BOOT-INF/lib/hadoop-hdfs-2.6.5.jar
BOOT-INF/lib/xercesImpl-2.9.1.jar
BOOT-INF/lib/xml-apis-1.4.01.jar
BOOT-INF/lib/htrace-core-3.0.4.jar
BOOT-INF/lib/hadoop-mapreduce-client-app-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-common-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-shuffle-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-core-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-jobclient-2.6.5.jar
BOOT-INF/lib/hadoop-annotations-2.6.5.jar
BOOT-INF/lib/spark-launcher_2.11-2.3.0.jar
BOOT-INF/lib/spark-kvstore_2.11-2.3.0.jar
BOOT-INF/lib/leveldbjni-all-1.8.jar
BOOT-INF/lib/jackson-core-2.9.6.jar
BOOT-INF/lib/spark-network-common_2.11-2.3.0.jar
BOOT-INF/lib/spark-network-shuffle_2.11-2.3.0.jar
BOOT-INF/lib/spark-unsafe_2.11-2.3.0.jar
BOOT-INF/lib/jets3t-0.9.4.jar
BOOT-INF/lib/httpcore-4.4.9.jar
BOOT-INF/lib/activation-1.1.1.jar
BOOT-INF/lib/bcprov-jdk15on-1.52.jar
BOOT-INF/lib/java-xmlbuilder-1.1.jar
BOOT-INF/lib/base64-2.3.8.jar
BOOT-INF/lib/curator-recipes-4.0.1.jar
BOOT-INF/lib/curator-framework-4.0.1.jar
BOOT-INF/lib/javax.servlet-api-3.1.0.jar
BOOT-INF/lib/commons-lang3-3.7.jar
BOOT-INF/lib/commons-math3-3.4.1.jar
BOOT-INF/lib/jsr305-1.3.9.jar
BOOT-INF/lib/slf4j-api-1.7.25.jar
BOOT-INF/lib/jul-to-slf4j-1.7.25.jar
BOOT-INF/lib/jcl-over-slf4j-1.7.25.jar
BOOT-INF/lib/compress-lzf-1.0.3.jar
BOOT-INF/lib/snappy-java-1.1.2.6.jar
BOOT-INF/lib/lz4-java-1.4.0.jar
BOOT-INF/lib/zstd-jni-1.3.2-2.jar
BOOT-INF/lib/RoaringBitmap-0.5.11.jar
BOOT-INF/lib/commons-net-2.2.jar
BOOT-INF/lib/scala-library-2.11.8.jar
BOOT-INF/lib/json4s-jackson_2.11-3.2.11.jar
BOOT-INF/lib/json4s-core_2.11-3.2.11.jar
BOOT-INF/lib/json4s-ast_2.11-3.2.11.jar
BOOT-INF/lib/scalap-2.11.0.jar
BOOT-INF/lib/scala-compiler-2.11.0.jar
BOOT-INF/lib/scala-xml_2.11-1.0.1.jar
BOOT-INF/lib/jersey-client-2.26.jar
BOOT-INF/lib/javax.ws.rs-api-2.1.jar
BOOT-INF/lib/javax.inject-2.5.0-b42.jar
BOOT-INF/lib/jersey-common-2.26.jar
BOOT-INF/lib/osgi-resource-locator-1.0.1.jar
BOOT-INF/lib/jersey-server-2.26.jar
BOOT-INF/lib/jersey-media-jaxb-2.26.jar
BOOT-INF/lib/validation-api-2.0.1.Final.jar
BOOT-INF/lib/jersey-container-servlet-2.26.jar
BOOT-INF/lib/jersey-container-servlet-core-2.26.jar
BOOT-INF/lib/netty-all-4.1.25.Final.jar
BOOT-INF/lib/netty-3.9.9.Final.jar
BOOT-INF/lib/stream-2.7.0.jar
BOOT-INF/lib/metrics-core-3.2.6.jar
BOOT-INF/lib/metrics-jvm-3.2.6.jar
BOOT-INF/lib/metrics-json-3.2.6.jar
BOOT-INF/lib/metrics-graphite-3.2.6.jar
BOOT-INF/lib/jackson-databind-2.9.6.jar
BOOT-INF/lib/jackson-module-scala_2.11-2.9.6.jar
BOOT-INF/lib/scala-reflect-2.11.11.jar
BOOT-INF/lib/jackson-module-paranamer-2.9.6.jar
BOOT-INF/lib/ivy-2.4.0.jar
BOOT-INF/lib/oro-2.0.8.jar
BOOT-INF/lib/pyrolite-4.13.jar
BOOT-INF/lib/py4j-0.10.6.jar
BOOT-INF/lib/spark-tags_2.11-2.3.0.jar
BOOT-INF/lib/commons-crypto-1.0.0.jar
BOOT-INF/lib/unused-1.0.0.jar
BOOT-INF/lib/simba-athena-jdbc-driver-2.0.2.jar
BOOT-INF/lib/spark-sql_2.11-2.3.0.jar
BOOT-INF/lib/univocity-parsers-2.5.9.jar
BOOT-INF/lib/spark-sketch_2.11-2.3.0.jar
BOOT-INF/lib/spark-catalyst_2.11-2.3.0.jar
BOOT-INF/lib/scala-parser-combinators_2.11-1.0.4.jar
BOOT-INF/lib/janino-3.0.8.jar
BOOT-INF/lib/commons-compiler-3.0.8.jar
BOOT-INF/lib/antlr4-runtime-4.7.jar
BOOT-INF/lib/orc-core-1.4.1-nohive.jar
BOOT-INF/lib/protobuf-java-2.5.0.jar
BOOT-INF/lib/commons-lang-2.6.jar
BOOT-INF/lib/aircompressor-0.8.jar
BOOT-INF/lib/orc-mapreduce-1.4.1-nohive.jar
BOOT-INF/lib/parquet-column-1.8.2.jar
BOOT-INF/lib/parquet-common-1.8.2.jar
BOOT-INF/lib/parquet-encoding-1.8.2.jar
BOOT-INF/lib/parquet-hadoop-1.8.2.jar
BOOT-INF/lib/parquet-format-2.3.1.jar
BOOT-INF/lib/parquet-jackson-1.8.2.jar
BOOT-INF/lib/arrow-vector-0.8.0.jar
BOOT-INF/lib/arrow-format-0.8.0.jar
BOOT-INF/lib/arrow-memory-0.8.0.jar
BOOT-INF/lib/hppc-0.7.2.jar
BOOT-INF/lib/flatbuffers-1.2.0-3f79e055.jar
BOOT-INF/lib/spark-yarn_2.11-2.3.0.jar
BOOT-INF/lib/hadoop-yarn-api-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-common-2.6.5.jar
BOOT-INF/lib/jaxb-api-2.3.0.jar
BOOT-INF/lib/jetty-util-6.1.26.jar
BOOT-INF/lib/jackson-jaxrs-1.9.13.jar
BOOT-INF/lib/jackson-xc-1.9.13.jar
BOOT-INF/lib/commons-cli-1.2.jar
BOOT-INF/lib/guice-servlet-3.0.jar
BOOT-INF/lib/commons-io-2.4.jar
BOOT-INF/lib/guice-3.0.jar
BOOT-INF/lib/javax.inject-1.jar
BOOT-INF/lib/aopalliance-1.0.jar
BOOT-INF/lib/log4j-1.2.17.jar
BOOT-INF/lib/hadoop-yarn-server-web-proxy-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-server-common-2.6.5.jar
BOOT-INF/lib/jetty-6.1.26.jar
BOOT-INF/lib/hadoop-yarn-client-2.6.5.jar
BOOT-INF/lib/hadoop-aws-2.7.3.jar
BOOT-INF/lib/hadoop-common-2.7.3.jar
BOOT-INF/lib/xmlenc-0.52.jar
BOOT-INF/lib/commons-collections-3.2.2.jar
BOOT-INF/lib/servlet-api-2.5.jar
BOOT-INF/lib/jsp-api-2.1.jar
BOOT-INF/lib/jersey-core-1.19.1.jar
BOOT-INF/lib/jsr311-api-1.1.1.jar
BOOT-INF/lib/jersey-json-1.9.jar
BOOT-INF/lib/jettison-1.1.jar
BOOT-INF/lib/jaxb-impl-2.2.3-1.jar
BOOT-INF/lib/jersey-server-1.19.1.jar
BOOT-INF/lib/commons-logging-1.1.3.jar
BOOT-INF/lib/commons-configuration-1.6.jar
BOOT-INF/lib/commons-digester-1.8.jar
BOOT-INF/lib/commons-beanutils-1.7.0.jar
BOOT-INF/lib/commons-beanutils-core-1.8.0.jar
BOOT-INF/lib/hadoop-auth-2.7.3.jar
BOOT-INF/lib/apacheds-kerberos-codec-2.0.0-M15.jar
BOOT-INF/lib/apacheds-i18n-2.0.0-M15.jar
BOOT-INF/lib/api-asn1-api-1.0.0-M20.jar
BOOT-INF/lib/api-util-1.0.0-M20.jar
BOOT-INF/lib/jsch-0.1.54.jar
BOOT-INF/lib/curator-client-2.7.1.jar
BOOT-INF/lib/htrace-core-3.1.0-incubating.jar
BOOT-INF/lib/zookeeper-3.4.6.jar
BOOT-INF/lib/jackson-annotations-2.9.0.jar
BOOT-INF/lib/aws-java-sdk-1.7.4.jar
BOOT-INF/lib/spark-hive_2.11-2.3.0.jar
BOOT-INF/lib/parquet-hadoop-bundle-1.6.0.jar
BOOT-INF/lib/hive-exec-1.2.1.spark2.jar
BOOT-INF/lib/javolution-5.5.1.jar
BOOT-INF/lib/apache-log4j-extras-1.2.17.jar
BOOT-INF/lib/antlr-runtime-3.4.jar
BOOT-INF/lib/stringtemplate-3.2.1.jar
BOOT-INF/lib/ST4-4.0.4.jar
BOOT-INF/lib/JavaEWAH-0.3.2.jar
BOOT-INF/lib/snappy-0.2.jar
BOOT-INF/lib/stax-api-1.0.1.jar
BOOT-INF/lib/opencsv-2.3.jar
BOOT-INF/lib/hive-metastore-1.2.1.spark2.jar
BOOT-INF/lib/bonecp-0.8.0.RELEASE.jar
BOOT-INF/lib/datanucleus-api-jdo-3.2.6.jar
BOOT-INF/lib/datanucleus-rdbms-3.2.9.jar
BOOT-INF/lib/commons-pool-1.6.jar
BOOT-INF/lib/commons-dbcp-1.4.jar
BOOT-INF/lib/jdo-api-3.0.1.jar
BOOT-INF/lib/jta-1.1.jar
BOOT-INF/lib/commons-httpclient-3.1.jar
BOOT-INF/lib/calcite-avatica-1.2.0-incubating.jar
BOOT-INF/lib/calcite-core-1.2.0-incubating.jar
BOOT-INF/lib/calcite-linq4j-1.2.0-incubating.jar
BOOT-INF/lib/eigenbase-properties-1.1.5.jar
BOOT-INF/lib/httpclient-4.5.5.jar
BOOT-INF/lib/jackson-mapper-asl-1.9.13.jar
BOOT-INF/lib/commons-codec-1.11.jar
BOOT-INF/lib/joda-time-2.9.9.jar
BOOT-INF/lib/jodd-core-3.5.2.jar
BOOT-INF/lib/datanucleus-core-3.2.10.jar
BOOT-INF/lib/libthrift-0.9.3.jar
BOOT-INF/lib/libfb303-0.9.3.jar
BOOT-INF/lib/derby-10.14.1.0.jar
BOOT-INF/lib/core-tech-starter-docker-spring-boot-1.0.0.jar

Lorsque vous essayez d'exécuter le

spark-submit --master yarn --deploy-mode client --driver-memory 10G --executor-memory 15G --class com.ati.media.ref.scd.ScheduleApplication

Je vois l'erreur suivante

java.lang.ClassNotFoundException: com.ati.media.ref.scd.ScheduleApplication
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Lorsque j'essaie d'utiliser

java -jar <jar> l'application se lance sans aucun problème.

java -cp résultat même erreur que spark-submit que calss principal non trouvé.

Ce qui ne va pas ici. J'ai la fonction principale dans ScheduleApplication .

Demandé el 18 de Avril, 2019 par rakeeee

Réponse

Trop de publicités?

1voto

Yosi Dahari avatar
Yosi Dahari Points 4476

Vous devez soit spécifier --jars <path to your jar> ou configurer l'emplacement des pots extrac au niveau du cluster

De la Documentation AWS :

Une action d'amorçage ou une configuration personnalisée remplace la classe de la classe. Lorsque cela se produit, le chargeur de classes ne récupère que les fichiers JAR qui existent dans l'emplacement que vous avez spécifié dans votre configuration.

Il y a peu d'options pour résoudre ce problème :

1) Spécifier le --jars avec un chemin vers votre bocal :

spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi --master yarn spark-examples.jar 100 --jars /home/hadoop/extrajars/*

2) ajouter le chemin de votre JAR personnalisé (contenant la classe manquante) au chemin des classes de Spark.

Sur /etc/spark/conf/spark-defaults.conf Dans l'exemple suivant, ajoutez le chemin d'accès de votre JAR personnalisé aux noms de classe spécifiés dans la trace de la pile d'erreurs. Dans l'exemple suivant, /home/hadoop/extrajars/* est le chemin du JAR personnalisé.

Par exemple (comment modifier un cluster en cours d'exécution) :

sudo vim /etc/spark/conf/spark-defaults.conf

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

Pour plus d'informations, ou pour savoir comment effectuer cette opération sur un nouveau cluster, reportez-vous à l'adresse suivante documentation

Répondu el 18 de Avril, 2019 par Yosi Dahari (4476 Points )

Questions en vedette

Top Tags

Dans notre réseau

Prograide.com

Prograide est une communauté de développeurs qui cherche à élargir la connaissance de la programmation au-delà de l'anglais.
Pour cela nous avons les plus grands doutes résolus en français et vous pouvez aussi poser vos propres questions ou résoudre celles des autres.

Powered by:

Yandex
X