Spark Connector: No results returned when invoking sqlContext.read.couchbase()

Spark Connector
1

I’m using spark connector 1.1.0 for scala 2.10. If I invoke sqlContext.read.couchbase() without the schemaFilter option inside couchbase(),like this:

val airline = sqlContext.read
.option(“bucket”,“travel-sample”)
.option(“schemaFilter”, "type = ‘airline’")
.couchbase()

I think it should be something like select * from travel-sample where type = ‘airline’, but in fact there will be no result returned because the N1QL generated is:
INFO N1QLRelation: Executing generated query: ‘SELECT FROM travel-sample WHERE null’.

2

@cjycjy so you are saying if you do it like

val airline = sql.read.couchbase(schemaFilter = EqualTo("type", "airline"))

it works, but when you do it like

val airline = sqlContext.read
.option("bucket","travel-sample")
.option("schemaFilter", "type = 'airline'")
.couchbase()

then the underlying N1QL query is generated wrongly and as a result you can’t get the records?

3

Exactly. couchase() will invoke buildFrame(null, null, None) and it will cover the original schemaFilter with null. So the underlying N1QL query will be wrong.

4

@cjycjy Are you using the Spark Connector with the Couchbase 4.5 developer preview? There have been some changes in the query API that we haven’t yet reflected in the Spark Connector. We typically wait until the product is fully baked before we put out compatible releases of the connectors. I’m not sure that’s what the problem is, but it’s possible.

5

No, My Couchbase version is 4.1.0.
I think it is a small bug in Spark Connector. The problem is that in the class N1QLRealation, when the queryFilter is constructed like

val queryFilter = if (parameters.get("schemaFilter").isDefined) { "WHERE " + parameters.get("schemaFilter").get } else { "" }

when the couchbase() is invoked with no schemaFilter, the (“schemaFilter”->null) will be set. So the if condition above will be true because it’s value is null.

6

I went ahead and filed a bug in Jira. Thanks for the report.

7

You are welcome.:wink: