Couchbase scala connector is having issues. It is using old core-io library 1.7.x. I am using couchbase scala sdk as well that is using new core-io 2.0.5. sbt is picking the latest one 2.0.5 that doesn’t have below method. Do we have any way to resolve this?
Exception in thread “main” java.lang.NoClassDefFoundError: com/couchbase/client/core/message/CouchbaseRequest
at com.couchbase.spark.rdd.QueryRDD.getPartitions(QueryRDD.scala:47)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:273)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:990)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
libraryDependencies in ThisBuild ++= Seq(
“com.couchbase.client” %% “scala-client” % “1.0.2”,
“org.apache.spark” %% “spark-core” % “2.4.5” ,
“org.apache.spark” %% “spark-streaming” % “2.4.5” ,
“org.apache.spark” %% “spark-sql” % “2.4.5” ,
“com.couchbase.client” %% “spark-connector” % “2.4.0”,
“org.scalaj” %% “scalaj-http” % “2.4.2”,
“com.eed3si9n” %% “gigahorse-asynchttpclient” % “0.5.0”,
“io.circe” %% “circe-core” % “0.10.0” % “optional”,
“org.scala-lang.modules” %% “scala-xml” % “1.2.0”,
“com.lihaoyi” %% “ujson” % “0.8.0”,
“com.lihaoyi” %% “upickle” % “0.8.0”,
“com.lihaoyi” %% “upack” % “0.8.0”,
“org.apache.lucene” % “lucene-core” % “6.6.2”,
“org.apache.lucene” % “lucene-misc” % “6.6.2”,
//“com.lucidworks.spark” % “spark-solr” % “3.6.6”
“org.apache.solr” % “solr-solrj” % “8.4.1”,
“io.ino” %% “solrs” % “2.4.1”
)
1 Like
Hey @eakarsu
Yes that’s correct, the Spark Connector is built using core-io 1.x, and is not compatible with the new Scala SDK which is based around a new core-io 2.x. So it’s not possible to combine the two in the same application, at this point. We do want to ultimately support the two together, most likely with a rebuild of the Spark Connector atop the Scala SDK, but I don’t have a timescale to offer for that work at this point.
Note that it is possible to do most operations with Key-Value, N1QL and FTS directly with the Spark Connector, without needing to use an SDK at all. There are many examples in the docs.
So the limitation may only be a problem if you
a) need to do something with the Spark Connector that it doesn’t support and are using the SDK to get around this (in which case, please let us know what is you’re doing, maybe I can suggest another way), or
b) you’re trying to do both Spark work and unrelated Couchbase operations in the same application, in which case splitting the app into two processes is the current workaround/solution
Graham,
Thanks for quick reply
I tried running couchbase spark connector only approach as well.
First of all, I am using ujson json library. I can not map spark results to ujson, it seems that I can do it to only JsonDocument.
sc.couchbaseGet[JsonDocument](Seq(“airline_10123”, “airline_10748”))
I want this
sc.couchbaseGet[ujson.Value](Seq(“airline_10123”, “airline_10748”))
Second, I was getting weird error: one jboss class is missing: org.jboss.netty.util.internal.ConcurrentHashMap
How can I resolve these?
Erol Akarsu
Yes, the contentAs[ujson.Value] functionality is only for the Scala SDK. (By the way, I’m glad to see that’s being used and is useful). Once we do the new version of the Spark Connector, it’s certainly something I’d be considering adding to it, but it’s not there currently. You’ll need to get it as a JsonDocument and then convert it to your desired representation.
On the class error, I don’t think that’s related to us? We do depend on netty, but we ‘shade’ (hide) that dependency, e.g. it appears under a com.couchbase namespace.