PROD URGENT: Multiple Errors and failures

Couchbase Server
, , ,
1

CB version: Couchbase Server Community Edition 7.1.1 build 3175

We have tried below

  1. Tested with 3 node Cluster and then on a Single node cluster.
  2. We have increased the Node memory on single node
    2.1 Data: 14000 MiB
    2.2 Index: 6000 MiB
    2.3: Search: 6000 MiB
  3. Increased Disk IOPS from 300 to 3000
  4. We have open all ports on (in security group)

We see below errors in logs
com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {“completed”:true,“coreId”:“0x57517b3100000001”,“errors”:[{“code”:5000,“message”:" read tcp 127.0.0.1:35824->127.0.0.1:9101: read: connection reset by peer from [127.0.0.1:9101] - cause: read tcp 127.0.0.1:35824->127.0.0.1:9101: read: connection reset by peer from [127.0.0.1:9101]“,“retry”:false}],“httpStatus”:500,“idempotent”:false,“lastDispatchedFrom”:”<>:44222",“lastDispatchedTo”:“<>:8093”,“requestId”:9538,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“28bead8b-bce3-4103-a4a4-b8600589e01c”,“statement”:"

6/09 07:47:52,246 ERROR [http-nio2-8080-exec-51] [uid:1fbfaf70cf5041d38df1d813b6fc59c9] PostDbSet: getTren
dingPlacesByMediaWithPublicContent()
com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {“completed
“:true,“coreId”:“0xd6cc8ab600000001”,“errors”:[{“code”:5000,“message”:” dial tcp 127.0.0.1:9101: connect: c
onnection refused from [127.0.0.1:9101] - cause: dial tcp 127.0.0.1:9101: connect: connection refused from
[127.0.0.1:9101]”,“retry”:false}],“httpStatus”:500,“idempotent”:false,"

: getTrendingFollowingPlaceIdsByUser()
com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {"completed
“:true,“coreId”:“0xd6cc8ab600000001”,“errors”:[{“code”:5000,“message”:” dial tcp 127.0.0.1:9101: connect: c
onnection refused from [127.0.0.1:9101]

t: getTrendingFollowingPlaceIdsByUser()
com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {“completed
“:true,“coreId”:“0xd6cc8ab600000001”,“errors”:[{“code”:5000,“message”:” dial tcp 127.0.0.1:9101: connect: c
onnection refused from [127.0.0.1:9101] - cause: dial tcp 127.0.0.1:9101: connect: connection refused from
[127.0.0.1:9101]”,“retry”:false}],“httpStatus”:500,“idempotent”:false,“lastDispatchedFrom”:“<>:
56574”,“lastDispatchedTo”:“<>:8093”

com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {“completed”:true,“coreId”:“0xd6cc8ab600000001”,“errors”:[{“code”:5000,“message”:" Index not ready for serving queries from [127.0.0.1:9101] - cause: Index not ready for serving queries from [127.0.0.1:9101]“,“retry”:false}],“httpStatus”:500,“idempotent”:false,“lastDispatchedFrom”:”<>:49006",“lastDispatchedTo”:“<>:8093”,“requestId”:353305,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“bdae9ab7-a939-49ac-8a59-c3fc227036d3”,“statement”:“SELECT f.* \nFROM fiestas f \nWHERE f.uid IN $userIds AND f.end IS MISSING AND f.tp = ‘f’ AND f.ftp = ‘man’ \n”,“type”:“query”},“timeoutMs”:30000,“timings”:{“dispatchMicros”:14059,“totalDispatchMicros”:14059,“totalMicros”:14446}}

com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {“completed”:true,“coreId”:“0xd6cc8ab600000001”,“errors”:[{“code”:5000,“message”:" dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101] - cause: dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101]“,“retry”:false}],“httpStatus”:500,“idempotent”:false,“lastDispatchedFrom”:”<>:55874",“lastDispatchedTo”:“<>:8093”,“requestId”:935,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“41df67f9-62b4-4235-97ab-fe6d4b9b2c30”,“statement”:

02/10 09:17:46,904 ERROR [http-nio2-8080-exec-119] [uid:1fbfaf70cf5041d38df1d813b6fc59c9] PlaceDbSet: getPlacesNearby()
com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error {“completed”:true,“coreId”:“0xe5c9b25200000001”,“errors”:[{“code”:5000,“message”:" runtime error: index out of range [0] with length 0 - cause: runtime error: index out of range [0] with length 0",“retry”:false}],“httpStatus”:200,“idempotent”:false,“lastDispatchedFrom”:“<>:59456”,“lastDispatchedTo”:“34.205.202.168:8093”,“requestId”:332,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“67012a7f-52dd-4900-b9db-748b534527f6”,“statement”:"

We have also tested this in Couchbase Server Enterprise Edition 7.1.1 build 3175 and everything works with no errors.

2

connection refused from [127.0.0.1:9101] - cause: dial tcp 127.0.0.1:9101: connect: connection refused from

Check indexer.log you might find useful info.

3

@vsr1 Any specific error should check for ?
@geraldss
Below are all the Errors in the logs
2022-10-02T10:03:41.655+00:00 [Info] SCAN#50 Error in getRequestedIndexSnapshot Index not ready for serving queries
2022-10-02T10:03:41.673+00:00 [Info] SCAN##53 Error in getRequestedIndexSnapshot Index not ready for serving queries
2022-10-02T10:03:48.627+00:00 [ERRO][FDB] Error in reading a doc block with id 1417125696632816 from a database file ‘/couchbase-data/@2i/places_idx_geo_location_3744681898615285168_0.index/data.fdb.4’
2022-10-02T10:03:48.627+00:00 [ERRO][FDB] Error in reading a doc length from offset 5804546853408014336 in block id 1417125696632816 from a database file ‘/couchbase-data/@2i/places_idx_geo_location_3744681898615285168_0.index/data.fdb.4’
2022-10-02T10:03:48.629+00:00 [ERRO][FDB] Error in reading the doc length metadata with offset 5804546853408014336 from a database file ‘/couchbase-data/@2i/places_idx_geo_location_3744681898615285168_0.index/data.fdb.4’
2022-10-02T10:04:06.432+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f. Err genServer.closed
2022-10-02T10:04:06.432+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received genServer.closed from 127.0.0.1:9999
2022-10-02T10:04:06.432+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f. Err projector.topicMissing
2022-10-02T10:04:06.432+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2022-10-02T10:09:06.349+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:36292. Error = EOF. Kill Pipe.
2022-10-02T10:09:06.349+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9100. Error = read tcp 127.0.0.1:36292->127.0.0.1:9100: use of closed network connection. Kill Pipe.
2022-10-02T10:14:06.350+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9100. Error = read tcp 127.0.0.1:37570->127.0.0.1:9100: use of closed network connection. Kill Pipe.
2022-10-02T10:14:06.350+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:37570. Error = EOF. Kill Pipe.
2022-10-02T10:19:06.348+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9100. Error = read tcp 127.0.0.1:38860->127.0.0.1:9100: use of closed network connection. Kill Pipe.
2022-10-02T10:19:06.348+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:38860. Error = EOF. Kill Pipe.
2022-10-02T10:20:52.892+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f. Err genServer.closed
2022-10-02T10:20:52.892+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received genServer.closed from 127.0.0.1:9999
2022-10-02T10:20:52.893+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f. Err projector.topicMissing
2022-10-02T10:20:52.893+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2022-10-02T10:20:54.336+00:00 [Info] SCAN##39 Error in getRequestedIndexSnapshot Index not ready for serving queries
2022-10-02T10:20:54.348+00:00 [Info] SCAN##40 Error in getRequestedIndexSnapshot Index not ready for serving queries
2022-10-02T10:25:52.780+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9100. Error = read tcp 127.0.0.1:41144->127.0.0.1:9100: use of closed network connection. Kill Pipe.
2022-10-02T10:25:52.780+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:41144. Error = EOF. Kill Pipe.
2022-10-02T10:27:23.916+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:40006. Error = EOF. Kill Pipe.
2022-10-02T10:27:23.925+00:00 [Error] DATP[->dataport “:9105”] worker “127.0.0.1:39218” exit: EOF
2022-10-02T10:27:23.925+00:00 [Error] DATP[->dataport “:9105”] remote “127.0.0.1:39218” closed
Received ConnectionError from Client for Stream MAINT_STREAM map[accounting:[171 516 515 635 773 968 292 870 301 606 477 988 126 891 608 826 376 594 703 74 556 621 809 273 296 533 549 855 691 95 431 40 300 1007 89 159 497 24 532 874 407 452 875 852 909 728 982 784 788 741 941 972 611 412 110 409 816 133 657 519 777 927 362 906 234 763 239 414 644 218 468 667 122 153 779 910 82 118 618 743 53 104 453 466 275 704 949 856 306 1018 165 155 863 879 346 518 827 706 215 222 399 547 560 928 527 108 730 824 504 119 1018 186 73 390 571 190 251 809 228 679 860 1015 905 756 901 44 536 613 467 340 790 894 75 226 772 116 642 981 774 554 259 507 656 324 787 907 157 276 140 247 644 130 484 813 915 980 529 25 992 162 178 887 132 601 211 450 562 720 32 512 596 548 607 117 452 359 859 59 856 602 306 719 236 726 871 526 7 187 70 342 947 535 543 811 104 195 281 988 492 495 916 154 626 883 876 142 95 661 432 739 989 476 924 934 297 481 534 345 235 616 713 62 271 681 693 749 457 692 898 1000 428 638 758 205 493 197 673 118 483 439 680 105 503 789 892 609 546 404 12 525 709 870 344 668 166 864 50 273 365 531 541 885 802 798 886 851 336 55 138 987 707 949 582 163 330 838 242 700 459 349 699 245 620 837 941 151 566 906 773 888 751 662 728 68 823 842 106 225 387 465 913 392 63 472 234 485 389 899 79 867 961 491 334 608 674 396 919 738 619 451 270 556 826 88 182 848 732 595 874 703 252 164 557 926 402 610 891 695 710 917 135 814 393 897 165 374 304 328 685 825 444 708 2 58 587 310 499 840 935 188 635 836 158 982 282 943]].
2022-10-02T10:27:23.932+00:00 [Info] TK ConnError MAINT_STREAM accounting [171 516 515 635 773 968 292 870 301 606 477 988 126 891 608 826 376 594 703 74 556 621 809 273 296 533 549 855 691 95 431 40 300 1007 89 159 497 24 532 874 407 452 875 852 909 728 982 784 788 741 941 972 611 412 110 409 816 133 657 519 777 927 362 906 234 763 239 414 644 218 468 667 122 153 779 910 82 118 618 743 53 104 453 466 275 704 949 856 306 1018 165 155 863 879 346 518 827 706 215 222 399 547 560 928 527 117 671 1009 1010 94 61 489 721 156 748 512 859 338 873 282 448 1002 99 945 63 637 898 781 97 280 822 840 915 28 158 631 715 946 805 531 764 263 423 487 834 553 668 617 403 558 559 587 897 180 226 450 123 16 397 670 575 427 585 602 685 510 186 634 699 752 889 580 150 419 511 473 692 966 391 317 147 358 564 930 1005 47 247 31 545 601 762 1021 212 374 503 79 25 835 451 839 841 0 129 934 148 814 639 41 360 960 540 340 689 255 919 294 633 19 87 732 13 4 651 348 262 384 371 711 49 293 463 265 543 570 323 387 593 493 62 1015 663 918 425 422 582 366 402 628 996 345 459 616 769 719 125 120 353 381 970 947 290 243 726 250 396 655 252 885 999 21 595 202 498 751 175 179 505 615 789 264 405 499 185 377 931 496 925 112 88 324 742 845 944 714 872 43 778 342 444 961 238 283 240 411 842 832 951 1003 50 565 978 164 388 160 191 911 167 609 124 625 329 850 7 439 193 509 15 551 797 901 449 584 1 629 775 355 882 908 106 343 481 725 550 599 34 812 401 22 72 1016 312 14 337 854 339 389 298 357 804 522 687 800 137 259 780 802 851 893 612 221 992 188 233 576 418 428 761 462 937 379 441 799 627 729 174 537 161 261 430 624 415 806 478 557 288 867 913 603 190 276 860 767 242 310 299 647 309 857 229 364 59 378 529 268 108 373 482 492 783 888 100 433 170 390 807 258 538 700 954 648 734 936 607 979 701 200 530 546 688 54 766 344 145 237 246 86 935 406 926 507 78 227 207 18 77 494 790 754 534 223 460 480 727 825 253 528 101 20 686 210 758 720 198 722 757 131 525 933 544 583 675 520 782 810 316 811 291 325 187 432 469 904 491 596 76 149 542 1008 184 853 508 922 152 178 1011 535 869 230 848 693 983 308 943 991 658 115 865 571 472 116 307 102 426 58 26 287 539 121 990 17 284 417 109 375 737 846 45 132 416 105 211 976 352 75 837 30 731 385 698 85 485 569 483 204 1012 201 479 959 341 907 656 46 713 476 1023 500 849 759 1014 695 836 3 981 315 796 562 182 504 67 470 787 69 436 662 55 623 676 144 445 232 372 332 682 969 42 216 438 199 57 958 66 84 488 642 442 486 745 517 437 581 645 998 44 957 638 923 801 335 948 277 896 217 1004 196 740 567 672 456 950 257 561 989 817 586 868 457 630 162 833 464 241 289 744 142 843 130 723 973 333 523 903 650 877 228 458 11 368 413 1001 181 334 214 354 828 285 626 673 718 113 260 168 392 410 717 818 351 443 995 613 68 755 1013 917 563 578 80 382 213 710 566 735 965 37 328 736 794 552 592 235 330 980 1019 597 632 661 318 111 916 620 829 1017 23 146 336 220 568 197 774 649 32 356 776 52 408 424 83 138 29 905 680 134 35 548 684 864 579 681 27 932 244 900 474 183 750 899 660 303 690 157 176 12 278 286 514 6 454 747 254 271 349 526 393 281 404 659 847 858 359 694 929 939 683 753 370 866 56 677 770 838 798 819 861 209 541 395 892 808 429 652 876 135 987 654 921 831 591 1006 33 71 231 249 434 697 709 862 997 461 942 792 93 598 60 166 64 880 952 139 940 386 9 363 963 313 573 696 984 883 554 795 484 5 975 821 114 521 733 641 48 471 350 679 749 361 39 136 446 467 669 793 455 302 938 962 924 365 666 2 588 665 270 813 977 555 305 985 786 844 881 322 383 266 953 224 830 81 319 653 993 590 600 219 369 192 245 746 614 765 107 674 643 163 956 205 577 705 730 820 1000 400 708 823 914 248 394 251 203 331 890 314 143 65 678 225 304 955 702 502 756 785 70 440 169 38 664 421 119 760 279 272 420 435 646 90 194 326 8 490 208 447 824 1020 506 140 884 622 791 269 610 572 605 895 871 974 103 127 894 902 36 51 716 295 513 536 971 589 886 967 172 189 619 771 73 173 347 495 177 321 154 887 311 195 640 724 297 707 465 475 320 398 96 712 986 91 236 327 878 141 604 739 501 772 912 98 1022 206 815 367 574 964 274 92 920 128 151 803 524 380 267 768 636 256 994 10 738]
2022-10-04T15:42:08.849+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f. Err projector.topicMissing
2022-10-04T15:42:08.849+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2022-10-04T15:42:08.850+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f. Err projector.topicMissing
2022-10-04T15:42:08.850+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2022-10-04T15:42:16.795+00:00 [Error] KVSender::sendMutationTopicRequest Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_ec20df92bb0ea97a478aef8b26b53f8f users
Unexpected Error feed.feeder
2022-10-04T15:42:16.795+00:00 [Error] KVSender::openMutationStream MAINT_STREAM users Error Received feed.feeder from 127.0.0.1:9999
2022-10-04T15:42:17.231+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.338+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.420+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.498+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.577+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.663+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.742+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.821+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.873+00:00 [Error] KVSender::openMutationStream MAINT_STREAM users Error from Projector feed.feeder
2022-10-04T15:42:17.873+00:00 [Error] Indexer::startKeyspaceIdStream Stream MAINT_STREAM KeyspaceId users SessionId 79411. Error from Projector feed.feeder. Retrying 1.
2022-10-04T15:42:17.899+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:17.973+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:18.048+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:18.049+00:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.123+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.207+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.283+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.358+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.429+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.504+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.585+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.655+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.760+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.835+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.916+00:00 [Error] GetMcConn(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:42:19.917+00:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg:
2022-10-04T15:47:09.556+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9100. Error = read tcp 127.0.0.1:41932->127.0.0.1:9100: use of closed network connection. Kill Pipe.
2022-10-04T15:47:09.556+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:41932. Error = EOF. Kill Pipe.
2022-10-04T15:52:09.554+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9100. Error = read tcp 127.0.0.1:43272->127.0.0.1:9100: use of closed network connection. Kill Pipe.
2022-10-04T15:52:09.554+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:43272. Error = EOF. Kill Pipe.

4

Hi @geraldss , Can you have a look at this, it is in production and these errors have started popping in production after our migration to 7.1 with the latest java sdk.

We are really running short of option. We have tried everything to rule out possible network or configuration issues. Opened all the ports for communication . Tried with a single node, tried with 3 nodes, tried in dev environment with smaller data set.

5

@Nikhil_Singh - please open a case with customer support where your issue can be given proper attention.

6

@sduvuru

Could the below FDB errors be due to auto-compaction?

The other errors on closeMutationStream/sendShutdownTopic are expected in some cases.

@Nikhil_Singh - Can you please share the complete set of logs so that we can look at what is happening in the cluster. You can upload them in a google drive & share the link over a DM.

Thanks,
Varun

7

This indicates a memory corruption. I can’t say for sure if auto compaction is the cause.
You can run forestdb_dump on the fdb file and see the disk image is corrupted.

8

Can you drop the index in question and re-create and see if that resolves.

9

I have sent the complete index logs in DM.

10

These error we are having on CE, as I have tried with Support they have replied that support is only for EE.

11

Hi we have tried this.
We have also dropped all indexes and re-created them at least 5 times but still see same issue.

12

If every time same index having issue? By any chance do u include array in that index for covering. If yes try remove array.

13

Create an account and open an issue at issues.couchbase.com

14

Created: Loading...

15

Hi @Nikhil_Singh ,

Thanks for sharing the logs. I see multiple issues in the logs:

a. Indexer panicked twice → We are not sure if this is due to ForestDb corruption. We will analyse the logs and come back to you on this.

b. Issue not getting resolved after all indexes were dropped and re-created. I do not see any “Indexer::handleDropIndex” messages in the logs you attached. Can you share the logs from the timeframe when the indexes where dropped and re-created but the issue still exists.

Thanks,
Varun