intermittently n1ql occurs error - {“msg”:“Index Not Found - cause: queryport.indexNotFound”,“code”:12016}
i use java sdk version 2.2.6.
a node excute well but others occurs error continuously when excute with cbq in each node.
Hi Donovan,
Can you send output of cbcollect_info. You can use the ‘log’ tab in web-console, or
http://developer.couchbase.com/documentation/server/4.1/cli/cbcollect-info-tool.html
Also, provide details of the setup, and exact steps to reproduce the issue.
thanks,
-Prasad
i sent you cbcollect files by message.
my setup is nothing special. three nodes in a cluster. version is 4.5 DP.
Could you elaborate on what you mean by “details of the setup”?
i upgraded to 4.5 beta.
but i still has problem.
i think it occured after recreating index.
i got a PoC under way.
it is so critical problem.
can i have some points for trouble shooting?
i think it is a java sdk problem.
i change the category.
because it is not occured in web ui.
@donovan.kim if you suspect it is a java SDK issue, can you please enable TRACE Logging and share the result when the issue happens? This allows us to look at the lower level network operations and get an idea whats sent and what comes back from the server. Also, please share the java code you are using to execute the query, thanks!
trace.log.zip (23.3 KB)
import static com.couchbase.client.java.query.Select.select;
import static com.couchbase.client.java.query.dsl.Alias.alias;
import static com.couchbase.client.java.query.dsl.Expression.path;
import static com.couchbase.client.java.query.dsl.Expression.s;
import static com.couchbase.client.java.query.dsl.Expression.x;
import static com.couchbase.client.java.query.dsl.functions.AggregateFunctions.max;
import static com.couchbase.client.java.query.dsl.functions.AggregateFunctions.min;
import static com.couchbase.client.java.query.dsl.functions.Case.caseSimple;
import static com.couchbase.client.java.query.dsl.functions.Collections.anyIn;
import com.couchbase.client.java.query.Statement;
import com.couchbase.client.java.query.dsl.Expression;
import com.couchbase.client.java.query.dsl.Sort;
import com.couchbase.client.java.query.dsl.path.FromPath;
import com.couchbase.client.java.query.dsl.path.HavingPath;
import com.couchbase.client.java.query.dsl.path.index.IndexReference;
import com.nsuslab.denma.common.service.pokercraft.UIConstants.GameKind;
import com.nsuslab.denma.rhyme.handhistory.storage.CouchbaseHandHistoryClient;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.eventbus.EventBus;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
public class SessionListWorker extends AbstractVerticle {
public static final String ADDRESS = "handhistory::getSessionList";
@Override
public void start() throws Exception {
EventBus eventBus = vertx.eventBus();
MessageConsumer<JsonObject> consumer = eventBus.localConsumer(ADDRESS);
consumer.handler(msg -> {
JsonObject param = msg.body();
int limit = param.getInteger("limit", 20);
int offset = param.getInteger("offset", 0);
String fromTime = getDefIfNull(param, "fromTime",
String.valueOf(System.currentTimeMillis() - 30 * 24 * 60 * 60 * 1000));//3 months
String toTime = getDefIfNull(param, "toTime", String.valueOf(System.currentTimeMillis()));
String gpid = param.getString("gpId");
String condition = param.getString("condition");
GameKind playType = GameKind.valueOf(getDefIfNull(param, "gameKind", "Holdem"));
final FromPath selectClause = select(
path("h", "tableId"),
path("p", "sessionId"),
max(path("h", "tableName")).as("tableName"),
min(path("h", "timestamp")).as("timestamp"),
max(path("h", "playType")).as("playType"),
max(path("h", "maxBuyIn")).as("maxBuyIn"),
max(path("h", "minBuyIn")).as("minBuyIn"),
max(path("h", "tourneyName")).as("tourneyName"),
max(path("h", "defaultBuyIn")).as("buyIn"),
max(path("h", "bigBlind")).as("bigBlind"),
max(path("h", "smallBlind")).as("smallBlind"));
HavingPath havingPath = selectClause
.from("HandHistory").as("h")
.useIndex(IndexReference.indexRef("HANDHISTORY_GPID_IDX"))
.unnest(path("h", "players")).as("p")
.where(buildCondition(fromTime, toTime, gpid, playType))
.groupBy(path("h", "tableId"), path("p", "sessionId"))
.letting(alias("time", min(path("h", "timestamp"))),
alias("cond", max(caseSimple(path("h", "playType"))
.when(s("3")).then(s("").concat(path("h", "defaultBuyIn")))
.when(s("2")).then(path("h", "tourneyName"))
.elseReturn(path("h", "smallBlind")
.concat(s("/"))
.concat(path("h", "bigBlind"))))));
Statement query = (condition == null ? havingPath : havingPath.having(x("cond").eq(s(condition))))
.orderBy(Sort.desc(x("time")))
.limit(limit)
.offset(offset);
try {
String result = CouchbaseHandHistoryClient.getInstance().executeQuery(query);
msg.reply(result);
} catch (Exception e) {
e.printStackTrace();
msg.fail(500, e.getMessage());
}
});
}
private Expression buildCondition(String fromTime, String toTime, String gpid, GameKind playType) {
Expression express = anyIn("t", path("h", "players")).satisfies(path("t", "userName").eq(s(gpid)))
.and(path("p", "userName").eq(s(gpid)))
.and(path("h", "timestamp").gte(s(fromTime)).and(path("h", "timestamp").lte(s(toTime))));
switch (playType) {
case Holdem:
express = express.and(path("h", "playType").eq(s("1")))
.and(path("h", "gameType").eq(s("0")))
.and(path("h", "gameBuyInType").ne(s("2")));
break;
case Omaha:
express = express.and(path("h", "playType").eq(s("1")))
.and(path("h", "gameType").eq(s("3")))
.and(path("h", "gameBuyInType").ne(s("2")));
break;
case AoF:
express = express.and(path("h", "playType").eq(s("1")))
.and(path("h", "gameBuyInType").eq(s("2")));
break;
case Tourney:
express = express.and(path("h", "playType").eq(s("2")));
break;
case SitAndGo:
express = express.and(path("h", "playType").eq(s("3")));
break;
default:
break;
}
return express;
}
private String getDefIfNull(JsonObject param, String key, String def) {
final String val = param.getString(key, def);
return val == null ? def : val;
}
@Override
public void stop() throws Exception {
}
}
public String executeQuery(final Statement stmt) {
final N1qlQueryResult result = executeStatement(_handhistoryBucketName, _handhistoryBucketPassword,
stmt);
final N1qlMetrics info = result.info();
final List<JsonObject> errors = result.errors();
errors.forEach(error -> {
System.out.println("^^^^^^^^^^^^^^^^^^^^^^^\n\n\n\n" + error.toString());
});
LOG.info(">>>> \n" + info);
final StringBuilder sb = new StringBuilder();
sb.append("[");
if (result.finalSuccess()) {
final List<N1qlQueryRow> allRows = result.allRows();
sb.append(allRows.stream().map(row -> row.value().toString()).collect(Collectors.joining(", ")));
}
sb.append("]");
return sb.toString();
}
protected N1qlQueryResult executeStatement(String bucketName, String password, Statement stmt) {
SimpleN1qlQuery simple = N1qlQuery.simple(stmt, N1qlParams.build().maxParallelism(12));
return executeQuery(bucketName, password, simple);
}
this problem looks disappeared.
i don’t know why.
is it possible?
i recreate the index about 8 hours ago.
if you see this in cbq as well, this is way more likely to be an indexer issue. @prasad were you able to make anything of the cbcollected logs?
it occured again after recreating index.
not web ui, but java sdk.
I am experiencing this same issue. Intermittent failures to find the index. Currently we have not yet scaled out the index nodes separately and the index is only available on a single node, but there is plenty of RAM available for the index on that node still and the node is not under a high load.
@zwood can you open a topic with all necessary details about your setup (CB version, nodes and cluster configuration, whether or not you’ve found hints in the server-side logs, etc…) in the Couchbase Server or N1QL section of the forums?
This is something reported directly by the N1QL service, not a bug/problem at the SDK level.
Thanks!
Simon,
Thanks for the response. I’ve opened a support request ticket and someone is looking into the issue. I have sent the Couchbase logs for that support request.
Dropping off on the forum now since the support request is in progress.
-Zach
Hi zwood,
what was resolution from couchbase support.
Any update on this issue? Facing the same problem here. (On 4.6.3-4136-E And 4.6.4-4590-E versions)
@krishan.jangid there are many different reasons why an index cannot be found. can you share more details of yours?
@daschl, thanks for quick response.
This issue is occurring on three different environments. Two of them have 4.6.4-4590 and one has 4.6.3-4136. The java sdk we are using is 2.5.5.
The query that is being executed was using intersect scan earlier because of non-covered indexes.
I created an index which improved the query execution time (prior to which the query was doing intersect scan). But this did not resolved this issue.
Later I found out that Index RAM that is being used is smaller than actual data size in RAM.
Before I changed the RAM size, the issue disappeared (on 4.6.4).
I have tried recreating the indexes as well but the issue still persists on 4.6.3-4136 but got fixed automatically on newer versions (I am using this word as I don’t know what fixed the issue).
So upgrading to a newer server version fixed the issue?