I am having an Json array imported into a single document in a bucket and i am trying to create dataframe from spark connector.
[{
“Name” :“Test”
“Salary”:2000
“Department”:“Maths”
},
{
“Name” :“Test”
“Salary”:2000
“Department”:“Maths”
}
]
I am using the code -
val conf = new SparkConf()
.setAppName(appName)
.set("spark.app.id", appName)
.set("spark.scheduler.mode", "FAIR")
.set("spark.couchbase.nodes", serverNameOrIP) // connect to Couchbase Server on localhost
.set("spark.couchbase.username", username) // with given credentials
.set("spark.couchbase.password", password)
conf.set("spark.couchbase.bucket." + bucket, "")
val spark = SparkSession.builder().config(conf).getOrCreate()
println("Default parallelism: " + spark.sparkContext.defaultParallelism)
//save stats
val sql = spark.sqlContext
import com.couchbase.spark.sql._
val bucket = bucketName
println("----Executing task on couchbase bucket--" + coll)
val dataFrame = sql.read.option("bucket", bucket).couchbase()
dataframe.printschema()
The dataframe i am getting is given below and 123 is Document ID
±------+
|META_ID|
±------+
| 123|
±------+
The schema is also not showing the elements and there are no array elements available in dataframe too