Azure Databricks and Couchbase Connectivity

Spark Connector
1

Hello,

I am trying to connect to Couchbase in an Azure Databricks (Spark) environment. However, I only see Couchbase Spark connectors for Spark 2.4 and Scala 2.12 OR Spark 2.3 and Scala 2.11.

Databricks only offers runtime environments for Spark 2.4 and Scala 2.11 OR Spark 3.0 and Scala 2.12.

Nevertheless, I have tried using both Couchbase connectors but ran into errors that I believe may be related to the versions differing.

For example, here is the code and error when using the Couchbase connector for Spark 2.4 and Scala 2.12 on a Databricks environment using Spark 2.4 and Scala 2.11 –

import com.couchbase.spark.
import com.couchbase.spark.sql.
import com.couchbase.client.java.document.JsonDocument
import org.apache.spark.sql.sources.EqualTo
import com.couchbase.client.java.query.N1qlQuery
import com.couchbase.client.java.view.ViewQuery

val sc = spark.sparkContext

sc.couchbaseGetJsonDocument.collect().foreach(println)

ERROR: java.lang.NoSuchMethodError: scala.Product.init(Lscala/Product;)V

And here are the spark config variables I’m setting in the databricks cluster –

spark.couchbase.password
spark.couchbase.username
spark.couchbase.nodes
spark.couchbase.bucket <bucket_name>

Our couchbase server is 6.0.1. Does anyone know if this is a version compatibility issue or if I am doing something wrong on my end?

Thank you for any help,
Zack

2

first of all sorry for the delay
I can answer for the first part of it
Spark 2.4 and Scala 2.12 thats on purpose 2.11 is EOL and we would want to look forward and so the connector is compatible only with Scala 2.12.
@graham.pople can you help assist with the other issue.

3

Hey @zsponaugle, sorry for the late response.

Yes, this configuration (Spark Connector built against Scala 2.12 being run in a 2.11 environment) won’t work. A NoSuchMethodError exception often indicates incompatible Scala versions, which is what’s happening here.

4

Hello,

Thank you both.

Would I also be correct in assuming that the older Couchbase 2.3 connector (spark 2.3 and scala 2.11) would NOT be compatible with the Databricks environment running Spark 2.4 and Scala 2.11?

Thanks,
Zack

5

@zsponaugle yes, unfortunately that is correct. We do not have - or plan to have - a Spark 2.4 connector that’s compatible with Scala 2.11.