Redact connection data

Spark Connector
1

In the logs can be seen:
couchbase://xaeslrba7b:xaeslrba7btest@couchbase01…

How is possible user and password to be redacted? I’ve tried with spark configuration properties: spark.sql.redaction.options.regex, spark.sql.redaction.string.regex and spark.secret.redactionPattern.

Please help.

2

How did you set the connection?

I would suggest to try something like that:

val spark = SparkSession.builder()
  .appName("Connection Redacted")
.
.
.
  .config("spark.sql.redaction.string.regex", ".*(couchbase://|couchbases://).*")
  .getOrCreate()
3

As I’ve written I’ve already tried that. I’ve tried it with spark.sql.redaction.string.regex, spark.redaction.regex and spark.redaction.string.regex. It doesn’t work.

4

An example of which logs you’re getting after the connection would help.

Another thing that I would suggest is to increase the log level on Spark.
Let’s assume that the connection logs are on the DEBUG level, you could switch to the INFO level, by using

spark.sparkContext.setLogLevel("INFO")
5

The log I’ve written is INFO level. Increase the log level won’t work. I don’t understand why the spark driver writes that with INFO level.

6

The next log is:
com.couchbase.client.core.cnc.LoggingEventConsumer [com.couchbase.transactions.cleanup][LogEvent] Waiting for 0 regular background threads to exit

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.