Impact of CoreEnvironment Static Block on Serverless Applications

Java SDK
1

Hi Couchbase Community,

I’m currently integrating Couchbase with a serverless application using Vert.x, and I came across a concern regarding the CoreEnvironment class in the Couchbase Java SDK.

The CoreEnvironment class has a static block that performs I/O operations to read and process all MANIFEST.MF files from the classpath

public class CoreEnvironment implements AutoCloseable {
    static {
        try {
            Enumeration<URL> resources = CoreEnvironment.class.getClassLoader().getResources(JarFile.MANIFEST_NAME);
            while (resources.hasMoreElements()) {
                URL manifestUrl = resources.nextElement();
                if (manifestUrl == null) {
                    continue;
                }
                Manifest manifest = new Manifest(manifestUrl.openStream());
                if (manifest.getEntries() == null) {
                    continue;
                }
                for (Map.Entry<String, Attributes> entry : manifest.getEntries().entrySet()) {
                    if (entry.getKey().startsWith("couchbase-")) {
                        MANIFEST_INFOS.put(entry.getKey(), entry.getValue());
                    }
                }
            }
        } catch (Exception e) {
            // Ignored on purpose.
        }
    }
}

From my observation, it takes around 4 seconds just to complete this operation when I first start the Verticle. This happens on my development laptop and our testing environment as well.
So I have a few questions

  1. What is the specific purpose of iterating through all JAR files and reading the manifest entries in the CoreEnvironment static block?
  2. Performing such extensive I/O operations during class loading seems to add significant startup latency and resource consumption. Does this approach align with the principles of serverless applications, where quick startup times and efficient resource usage are critical?

Any insights or suggestions on how to better handle this scenario would be greatly appreciated.

Thanks in advance!

1 Like
2

I opened a ticket Loading...

It gets the version and git commit from the manifest.

1 Like
3

I put some timing in CoreEnvironment. That static block takes 11ms in my environment. Initializing SLF4J appears to take about 600ms. The majority of the rest of the time (another 600ms) seems to be in ioEnvironment.build() - which creates several event loops and initializes netty.

If you could take some thread dumps during initialization in your environment, that would help us figure out why it is taking so long.

4

Hi Ethan,

We can try using a different strategy to get the version info. Maybe it will help.

In the meantime, can you confirm the delay is not coming from Log4j2? See java - log4j (2.20.0) takes several seconds to initialize - Stack Overflow

Thanks,
David

5

Hi Michael,

I rerun the code again in Springboot 3 with some random libraries just to see how it perform compared to our in-house reactive framework built on top of Vertx. It did take 350ms to run create ClusterEnvironment with builder on my decent specs laptop especially with medium size fat jar. This is kinda misleading because we all expect the builder to execute almost instantly.

To reproduce this issue, we can run this code as part of any standalone executable fat jar. The key observation here is that the more 3rd party jar files included in the final fat jar, the longer it take to scan all the jar files (in this case Spring Boot, I’ll retry with Vertx)

Here’s the code I used for this Spring boot testing. It would be interesting to compare how it performs in your environment.

When you try it, kindly execute as a standalone jar file (java -jar) instead of using IDE like InteliJ/Eclipse

2024-08-05T20:35:26.947-04:00 INFO 37636 — [ main] com.example.demo.DemoApplication : Creating ClusterEnvironment
2024-08-05T20:35:27.250-04:00 INFO 37636 — [ main] com.example.demo.DemoApplication : ClusterEnvironment created

Thanks

6

Hi David,
Kindly try the example above and let me know, there should be logback but not log4j2. The impact isn’t as significant but I’m wondering should it take instantly just to create new ClusterEnvinroment instance?

Thanks,

7

Hi Ethan,

Oh! Thanks for sharing that demo project. What you’re measuring is the time it takes the JVM to load all the classes used by the SDK. Not manifest scanning, just plain old Java class loading.

The first time I create a ClusterEnvironment in your demo project, it takes hundreds of milliseconds. If I immediately create a second one, it takes just 1 millisecond.

To verify this is nothing to do with manifest scanning, I tested a custom-built SDK that skips all manifest scanning, and got similar results.

I’m not sure what we could do to reduce the class loading time – this is just the JVM being the JVM.

Thanks,
David

8

That said, we’ll still look into alternatives to manifest scanning under the JVMCBC-1554 ticket Mike created.

9

Hi David,

Thanks for the update. I addeds some screenshots in the same git repo

Oh! Thanks for sharing that demo project. What you’re measuring is the time it takes the JVM to load all the classes used by the SDK. Not manifest scanning, just plain old Java class loading.

The measurement I did is before and after ClusterEnvironment creation. The entire spring boot app need more than that to load all classes & create application context (1+ seconds on my personal laptop)

The first time I create a ClusterEnvironment in your demo project, it takes hundreds of milliseconds. If I immediately create a second one, it takes just 1 millisecond.

I agree with you on this. Because that static block is also only execute once when classes loading happen in JVM.
But then this will be execute multiple times because of the nature of serverless application (AWS Lambda cold start for example), the first request always take a hit in term of response time. And when the function idle, it gets undeployed. The next time when new request come, the same initialization happen again…

To verify this is nothing to do with manifest scanning, I tested a custom-built SDK that skips all manifest scanning, and got similar results.

I’ve discoverred that this manifest scanning actually scanning 100+ jar files in Spring boot example I shared. This operation takes 300ms in my company laptop.
With our in-house framework, this scans 1000+ jar files, which took 3-5s on the same company laptop.
This is partially our own problem because the way our framework works. And I’m trying to find a workaround.

custom-built SDK that skips all manifest scanning.

Is this something can be shared. Or if you can give us some instruction to replicate this custom SDK, that would be greatly appreciated! I can use it for testing and get back to you on the result.

10

custom-built SDK that skips all manifest scanning.

You can clone the repository, follow the instructions in the README to build and install the -SNAPSHOT into your local .m2 repository and use that.

Modifiy CoreEnvironment by commenting out the code that scans the manifests.

1 Like
11

Having a hard time building the root module because of protostellar, unittest failure & java-doc mvn plugin . (with git tag core-io-2.6.2)
I was able to build some individual modules in my mac by changing some mvn plugin config: protostellar, test-util, CouchbaseMock, core-io-deps → core-io-2.6.2 (with MANIFEST code remove)
I could see noticable improvement, 4s goes down to 2s now for creating ClusterEnvironment. But there’s seems to be other classes doing the same thing. Doing some more testing.

12

Follow the Readme. Use the Makefile.

$ git clone https://github.com/couchbase/couchbase-jvm-clients.git
$ cd couchbase-jvm-clients
$ make

seems to be other classes doing the same thing.

Other classes reading manifests?

1 Like
13

Thanks for the update. That’s great news. In a few days we’ll hopefully be able to share a 3.7.3-SNAPSHOT that uses a more efficient strategy to read the component versions.

We’re happy to help you get a clean build. If you like, please share the commands you’re using to build the SDK and the errors you’re seeing. As long as you’re using JDK 17 or later and running make as Mike recommended (or running the commands in the Makefile manually) everything should go smoothly.

Thanks,
David

1 Like
14

Hi Micheal, David,
Thanks for sharing the steps. There’s some issue with make cmd in my company laptop behind corporate proxy. Pretty sure it will work with personal devices.

couchbase-jvm-clients % make -d
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Reading makefile `Makefile'...
Updating makefiles....
 Considering target file `Makefile'.
  Looking for an implicit rule for `Makefile'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.o'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.c'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.cc'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.C'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.cpp'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.p'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.f'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.F'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.m'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.r'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.s'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.S'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.mod'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.sh'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile,v'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `RCS/Makefile,v'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `RCS/Makefile'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `s.Makefile'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `SCCS/s.Makefile'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.o'.
  Looking for a rule with intermediate file `Makefile.o'.
   Avoiding implicit rule recursion.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.c'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.cc'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.C'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.cpp'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.p'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.f'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.F'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.m'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.r'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.s'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.S'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.mod'.
   Trying pattern rule with stem `Makefile.o'.
   Trying implicit prerequisite `Makefile.o,v'.
   Trying pattern rule with stem `Makefile.o'.
   Trying implicit prerequisite `RCS/Makefile.o,v'.
   Trying pattern rule with stem `Makefile.o'.
   Trying implicit prerequisite `RCS/Makefile.o'.
   Trying pattern rule with stem `Makefile.o'.
   Trying implicit prerequisite `s.Makefile.o'.
   Trying pattern rule with stem `Makefile.o'.
   Trying implicit prerequisite `SCCS/s.Makefile.o'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.c'.
   Looking for a rule with intermediate file `Makefile.c'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.y'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.l'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.w'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.w'.
    Trying pattern rule with stem `Makefile.c'.
    Trying implicit prerequisite `Makefile.c,v'.
    Trying pattern rule with stem `Makefile.c'.
    Trying implicit prerequisite `RCS/Makefile.c,v'.
    Trying pattern rule with stem `Makefile.c'.
    Trying implicit prerequisite `RCS/Makefile.c'.
    Trying pattern rule with stem `Makefile.c'.
    Trying implicit prerequisite `s.Makefile.c'.
    Trying pattern rule with stem `Makefile.c'.
    Trying implicit prerequisite `SCCS/s.Makefile.c'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.y'.
    Looking for a rule with intermediate file `Makefile.y'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.y'.
     Trying implicit prerequisite `Makefile.y,v'.
     Trying pattern rule with stem `Makefile.y'.
     Trying implicit prerequisite `RCS/Makefile.y,v'.
     Trying pattern rule with stem `Makefile.y'.
     Trying implicit prerequisite `RCS/Makefile.y'.
     Trying pattern rule with stem `Makefile.y'.
     Trying implicit prerequisite `s.Makefile.y'.
     Trying pattern rule with stem `Makefile.y'.
     Trying implicit prerequisite `SCCS/s.Makefile.y'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.l'.
    Looking for a rule with intermediate file `Makefile.l'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.l'.
     Trying implicit prerequisite `Makefile.l,v'.
     Trying pattern rule with stem `Makefile.l'.
     Trying implicit prerequisite `RCS/Makefile.l,v'.
     Trying pattern rule with stem `Makefile.l'.
     Trying implicit prerequisite `RCS/Makefile.l'.
     Trying pattern rule with stem `Makefile.l'.
     Trying implicit prerequisite `s.Makefile.l'.
     Trying pattern rule with stem `Makefile.l'.
     Trying implicit prerequisite `SCCS/s.Makefile.l'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.w'.
    Looking for a rule with intermediate file `Makefile.w'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.w'.
     Trying implicit prerequisite `Makefile.w,v'.
     Trying pattern rule with stem `Makefile.w'.
     Trying implicit prerequisite `RCS/Makefile.w,v'.
     Trying pattern rule with stem `Makefile.w'.
     Trying implicit prerequisite `RCS/Makefile.w'.
     Trying pattern rule with stem `Makefile.w'.
     Trying implicit prerequisite `s.Makefile.w'.
     Trying pattern rule with stem `Makefile.w'.
     Trying implicit prerequisite `SCCS/s.Makefile.w'.
    Trying pattern rule with stem `Makefile'.
    Rejecting impossible implicit prerequisite `Makefile.w'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.cc'.
   Looking for a rule with intermediate file `Makefile.cc'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile.cc'.
    Trying implicit prerequisite `Makefile.cc,v'.
    Trying pattern rule with stem `Makefile.cc'.
    Trying implicit prerequisite `RCS/Makefile.cc,v'.
    Trying pattern rule with stem `Makefile.cc'.
    Trying implicit prerequisite `RCS/Makefile.cc'.
    Trying pattern rule with stem `Makefile.cc'.
    Trying implicit prerequisite `s.Makefile.cc'.
    Trying pattern rule with stem `Makefile.cc'.
    Trying implicit prerequisite `SCCS/s.Makefile.cc'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.C'.
   Looking for a rule with intermediate file `Makefile.C'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile.C'.
    Trying implicit prerequisite `Makefile.C,v'.
    Trying pattern rule with stem `Makefile.C'.
    Trying implicit prerequisite `RCS/Makefile.C,v'.
    Trying pattern rule with stem `Makefile.C'.
    Trying implicit prerequisite `RCS/Makefile.C'.
    Trying pattern rule with stem `Makefile.C'.
    Trying implicit prerequisite `s.Makefile.C'.
    Trying pattern rule with stem `Makefile.C'.
    Trying implicit prerequisite `SCCS/s.Makefile.C'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.cpp'.
   Looking for a rule with intermediate file `Makefile.cpp'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile.cpp'.
    Trying implicit prerequisite `Makefile.cpp,v'.
    Trying pattern rule with stem `Makefile.cpp'.
    Trying implicit prerequisite `RCS/Makefile.cpp,v'.
    Trying pattern rule with stem `Makefile.cpp'.
    Trying implicit prerequisite `RCS/Makefile.cpp'.
    Trying pattern rule with stem `Makefile.cpp'.
    Trying implicit prerequisite `s.Makefile.cpp'.
    Trying pattern rule with stem `Makefile.cpp'.
    Trying implicit prerequisite `SCCS/s.Makefile.cpp'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.p'.
   Looking for a rule with intermediate file `Makefile.p'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.web'.
    Trying pattern rule with stem `Makefile.p'.
    Trying implicit prerequisite `Makefile.p,v'.
    Trying pattern rule with stem `Makefile.p'.
    Trying implicit prerequisite `RCS/Makefile.p,v'.
    Trying pattern rule with stem `Makefile.p'.
    Trying implicit prerequisite `RCS/Makefile.p'.
    Trying pattern rule with stem `Makefile.p'.
    Trying implicit prerequisite `s.Makefile.p'.
    Trying pattern rule with stem `Makefile.p'.
    Trying implicit prerequisite `SCCS/s.Makefile.p'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.web'.
    Looking for a rule with intermediate file `Makefile.web'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.web'.
     Trying implicit prerequisite `Makefile.web,v'.
     Trying pattern rule with stem `Makefile.web'.
     Trying implicit prerequisite `RCS/Makefile.web,v'.
     Trying pattern rule with stem `Makefile.web'.
     Trying implicit prerequisite `RCS/Makefile.web'.
     Trying pattern rule with stem `Makefile.web'.
     Trying implicit prerequisite `s.Makefile.web'.
     Trying pattern rule with stem `Makefile.web'.
     Trying implicit prerequisite `SCCS/s.Makefile.web'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.f'.
   Looking for a rule with intermediate file `Makefile.f'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.F'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.r'.
    Trying pattern rule with stem `Makefile.f'.
    Trying implicit prerequisite `Makefile.f,v'.
    Trying pattern rule with stem `Makefile.f'.
    Trying implicit prerequisite `RCS/Makefile.f,v'.
    Trying pattern rule with stem `Makefile.f'.
    Trying implicit prerequisite `RCS/Makefile.f'.
    Trying pattern rule with stem `Makefile.f'.
    Trying implicit prerequisite `s.Makefile.f'.
    Trying pattern rule with stem `Makefile.f'.
    Trying implicit prerequisite `SCCS/s.Makefile.f'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.F'.
    Looking for a rule with intermediate file `Makefile.F'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.F'.
     Trying implicit prerequisite `Makefile.F,v'.
     Trying pattern rule with stem `Makefile.F'.
     Trying implicit prerequisite `RCS/Makefile.F,v'.
     Trying pattern rule with stem `Makefile.F'.
     Trying implicit prerequisite `RCS/Makefile.F'.
     Trying pattern rule with stem `Makefile.F'.
     Trying implicit prerequisite `s.Makefile.F'.
     Trying pattern rule with stem `Makefile.F'.
     Trying implicit prerequisite `SCCS/s.Makefile.F'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.r'.
    Looking for a rule with intermediate file `Makefile.r'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile'.
     Rejecting impossible implicit prerequisite `Makefile.l'.
     Trying pattern rule with stem `Makefile.r'.
     Trying implicit prerequisite `Makefile.r,v'.
     Trying pattern rule with stem `Makefile.r'.
     Trying implicit prerequisite `RCS/Makefile.r,v'.
     Trying pattern rule with stem `Makefile.r'.
     Trying implicit prerequisite `RCS/Makefile.r'.
     Trying pattern rule with stem `Makefile.r'.
     Trying implicit prerequisite `s.Makefile.r'.
     Trying pattern rule with stem `Makefile.r'.
     Trying implicit prerequisite `SCCS/s.Makefile.r'.
   Trying pattern rule with stem `Makefile'.
   Rejecting impossible implicit prerequisite `Makefile.F'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.m'.
   Looking for a rule with intermediate file `Makefile.m'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.ym'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.lm'.
    Trying pattern rule with stem `Makefile.m'.
    Trying implicit prerequisite `Makefile.m,v'.
    Trying pattern rule with stem `Makefile.m'.
    Trying implicit prerequisite `RCS/Makefile.m,v'.
    Trying pattern rule with stem `Makefile.m'.
    Trying implicit prerequisite `RCS/Makefile.m'.
    Trying pattern rule with stem `Makefile.m'.
    Trying implicit prerequisite `s.Makefile.m'.
    Trying pattern rule with stem `Makefile.m'.
    Trying implicit prerequisite `SCCS/s.Makefile.m'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.ym'.
    Looking for a rule with intermediate file `Makefile.ym'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.ym'.
     Trying implicit prerequisite `Makefile.ym,v'.
     Trying pattern rule with stem `Makefile.ym'.
     Trying implicit prerequisite `RCS/Makefile.ym,v'.
     Trying pattern rule with stem `Makefile.ym'.
     Trying implicit prerequisite `RCS/Makefile.ym'.
     Trying pattern rule with stem `Makefile.ym'.
     Trying implicit prerequisite `s.Makefile.ym'.
     Trying pattern rule with stem `Makefile.ym'.
     Trying implicit prerequisite `SCCS/s.Makefile.ym'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.lm'.
    Looking for a rule with intermediate file `Makefile.lm'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.lm'.
     Trying implicit prerequisite `Makefile.lm,v'.
     Trying pattern rule with stem `Makefile.lm'.
     Trying implicit prerequisite `RCS/Makefile.lm,v'.
     Trying pattern rule with stem `Makefile.lm'.
     Trying implicit prerequisite `RCS/Makefile.lm'.
     Trying pattern rule with stem `Makefile.lm'.
     Trying implicit prerequisite `s.Makefile.lm'.
     Trying pattern rule with stem `Makefile.lm'.
     Trying implicit prerequisite `SCCS/s.Makefile.lm'.
   Trying pattern rule with stem `Makefile'.
   Rejecting impossible implicit prerequisite `Makefile.r'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.s'.
   Looking for a rule with intermediate file `Makefile.s'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.S'.
    Trying pattern rule with stem `Makefile.s'.
    Trying implicit prerequisite `Makefile.s,v'.
    Trying pattern rule with stem `Makefile.s'.
    Trying implicit prerequisite `RCS/Makefile.s,v'.
    Trying pattern rule with stem `Makefile.s'.
    Trying implicit prerequisite `RCS/Makefile.s'.
    Trying pattern rule with stem `Makefile.s'.
    Trying implicit prerequisite `s.Makefile.s'.
    Trying pattern rule with stem `Makefile.s'.
    Trying implicit prerequisite `SCCS/s.Makefile.s'.
    Trying pattern rule with stem `Makefile'.
    Trying implicit prerequisite `Makefile.S'.
    Looking for a rule with intermediate file `Makefile.S'.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Avoiding implicit rule recursion.
     Trying pattern rule with stem `Makefile.S'.
     Trying implicit prerequisite `Makefile.S,v'.
     Trying pattern rule with stem `Makefile.S'.
     Trying implicit prerequisite `RCS/Makefile.S,v'.
     Trying pattern rule with stem `Makefile.S'.
     Trying implicit prerequisite `RCS/Makefile.S'.
     Trying pattern rule with stem `Makefile.S'.
     Trying implicit prerequisite `s.Makefile.S'.
     Trying pattern rule with stem `Makefile.S'.
     Trying implicit prerequisite `SCCS/s.Makefile.S'.
   Trying pattern rule with stem `Makefile'.
   Rejecting impossible implicit prerequisite `Makefile.S'.
   Trying pattern rule with stem `Makefile'.
   Trying implicit prerequisite `Makefile.mod'.
   Looking for a rule with intermediate file `Makefile.mod'.
    Avoiding implicit rule recursion.
    Avoiding implicit rule recursion.
    Trying pattern rule with stem `Makefile.mod'.
    Trying implicit prerequisite `Makefile.mod,v'.
    Trying pattern rule with stem `Makefile.mod'.
    Trying implicit prerequisite `RCS/Makefile.mod,v'.
    Trying pattern rule with stem `Makefile.mod'.
    Trying implicit prerequisite `RCS/Makefile.mod'.
    Trying pattern rule with stem `Makefile.mod'.
    Trying implicit prerequisite `s.Makefile.mod'.
    Trying pattern rule with stem `Makefile.mod'.
    Trying implicit prerequisite `SCCS/s.Makefile.mod'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.c'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.cc'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.C'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.cpp'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.p'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.f'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.F'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.m'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.r'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.s'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.S'.
  Trying pattern rule with stem `Makefile'.
  Rejecting impossible implicit prerequisite `Makefile.mod'.
  Trying pattern rule with stem `Makefile'.
  Trying implicit prerequisite `Makefile.sh'.
  Looking for a rule with intermediate file `Makefile.sh'.
   Avoiding implicit rule recursion.
   Trying pattern rule with stem `Makefile.sh'.
   Trying implicit prerequisite `Makefile.sh,v'.
   Trying pattern rule with stem `Makefile.sh'.
   Trying implicit prerequisite `RCS/Makefile.sh,v'.
   Trying pattern rule with stem `Makefile.sh'.
   Trying implicit prerequisite `RCS/Makefile.sh'.
   Trying pattern rule with stem `Makefile.sh'.
   Trying implicit prerequisite `s.Makefile.sh'.
   Trying pattern rule with stem `Makefile.sh'.
   Trying implicit prerequisite `SCCS/s.Makefile.sh'.
  No implicit rule found for `Makefile'.
  Finished prerequisites of target file `Makefile'.
 No need to remake target `Makefile'.
Updating goal targets....
Considering target file `install'.
 File `install' does not exist.
  Considering target file `deps-only'.
   File `deps-only' does not exist.
   Finished prerequisites of target file `deps-only'.
  Must remake target `deps-only'.
Putting child 0x600000590050 (deps-only) PID 72054 on the chain.
Live child 0x600000590050 (deps-only) PID 72054 


Exception in thread "main" java.net.ConnectException: Operation timed out (Connection timed out)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:615)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1192)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1086)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1619)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1547)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
	at org.apache.maven.wrapper.DefaultDownloader.downloadInternal(DefaultDownloader.java:95)
	at org.apache.maven.wrapper.DefaultDownloader.download(DefaultDownloader.java:85)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:84)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:160)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:73)
Reaping losing child 0x600000590050 PID 72054 
make: *** [deps-only] Error 1
Removing child 0x600000590050 PID 72054 from chain.

Yes, OpenTelemetryMeter and OpenTelemetryRequestTracer.

Thanks for the update. That’s great news. In a few days we’ll hopefully be able to share a 3.7.3-SNAPSHOT that uses a more efficient strategy to read the component versions.

If the snapshot version could be published to public maven repository, that would be greatly appreciated! It’s not straightforward to create my own couchbase client build & publish to our internal artifactory for higher environment testing.
We’re using Couchbase server 6.6 so I hope they’re still compatible.

1 Like
15

At point where you get “Operation timed out (Connection timed out)”, it should be downloading from /repo.maven.apache.org. Maybe you have network connectivity issues? A proxy configuration maybe?
Does this command execute successfully?
curl https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/maven-metadata.xml -o curl.out

Must remake target `deps-only'.
Putting child 0x600001e9c050 (deps-only) PID 53675 on the chain.
Live child 0x600001e9c050 (deps-only) PID 53675
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< com.couchbase.client:protostellar >------------------
[INFO] Building Couchbase Protostellar GRPC 9999.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- clean:3.2.0:clean (default-clean) @ protostellar ---
[INFO] Deleting /Users/michaelreiche/couchbase-jvm-clients/protostellar/target
[INFO] Deleting /Users/michaelreiche/couchbase-jvm-clients/protostellar/src (includes = [], excludes = [])
[INFO]
[INFO] --- protoc-jar:3.11.4:run (default) @ protostellar ---
[INFO] Protoc version: 3.23.2
protoc-jar: protoc version: 3.23.2, detected platform: osx-x86_64 (mac os x/x86_64)
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: saved: /var/folders/16/lm68ltkx44q40xpvtd6w585r0000gp/T/protocjar.webcache/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: cached: /var/folders/16/lm68ltkx44q40xpvtd6w585r0000gp/T/protocjar.webcache/com/google/protobuf/protoc/3.23.2/protoc-3.23.2-osx-x86_64.exe
protoc-jar: executing: [/var/folders/16/lm68ltkx44q40xpvtd6w585r0000gp/T/protocjar7036952151954099226/bin/protoc.exe, --version]
libprotoc 23.2
16

Not sure what’s going on with make there. But if you have a look at the Makefile it’s just executing 3 lines that you can run outside of make. It just builds and installs some dependencies into your Maven local repo, to workaround Maven not having native support for this.

17

When the Gerrit change in the ticket is merged, the snapshot will be built and published.
Use this in your POM when building to get the SNAPSHOT.

       <repository>
         <id>snapshots-repo</id>
         <url>https://oss.sonatype.org/content/repositories/snapshots</url>
         <releases><enabled>false</enabled></releases>
         <snapshots><enabled>true</enabled></snapshots>
       </repository>
     </repositories>
2 Likes
18

At point where you get “Operation timed out (Connection timed out)”, it should be downloading from /repo.maven.apache.org. Maybe you have network connectivity issues? A proxy configuration maybe?

Not sure what’s going on with make there. But if you have a look at the Makefile it’s just executing 3 lines that you can run outside of make. It just builds and installs some dependencies into your Maven local repo, to workaround Maven not having native support for this.

Yes, as shared earlier, my laptop is configured to run behind corporate proxy and artifactory. If I updated the makefile to use my preconfigured maven instead of wrapper. Getting below error.

make                                                                                               
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< com.couchbase.client:protostellar >------------------
[INFO] Building Couchbase Protostellar GRPC 9999.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- clean:3.2.0:clean (default-clean) @ protostellar ---
[INFO] 
[INFO] --- protoc-jar:3.11.4:run (default) @ protostellar ---
[INFO] Protoc version: 3.23.2
protoc-jar: protoc version: 3.23.2, detected platform: osx-aarch_64 (mac os x/aarch64)
protoc-jar: using default maven settings, didn't find user settings.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: using default maven settings, didn't find user settings.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/github/os72/protoc/maven-metadata.xml
protoc-jar: using default maven settings, didn't find user settings.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: using default maven settings, didn't find user settings.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/github/os72/protoc/maven-metadata.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20.493 s
[INFO] Finished at: 2024-08-08T13:42:35-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project protostellar: Error extracting protoc for version 3.23.2: Unsupported platform: protoc-3.23.2-osx-aarch_64.exe -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
make: *** [deps-only] Error 1

I have fixed this previously by adding.

                <artifactId>protoc-jar-maven-plugin</artifactId>
                <version>3.11.4</version>
                <configuration>
                    <protocArtifact>com.google.protobuf:protoc:3.21.1</protocArtifact>
                </configuration>

There was also some issue with javadocs but I can’t recall the details. In the end, I was able to build the core-io jar by building individual modules as follow: protostellar, test-util, CouchbaseMock, core-io-deps then finnaly core-io-2.6.2 (with MANIFEST code removed)
Let’s assume I can make the build. Testing on local envinronment confirms improvement.

When the Gerrit change in the ticket is merged, the snapshot will be built and published.
Use this in your POM when building to get the SNAPSHOT.

Good to hear that! Guess now I can just wait for the snapshot being published then I can move on with higher environment testing. Thank you!

19

This should also work for M1.

<groupId>io.github.blackrock</groupId>
<artifactId>protoc-maven-plugin</artifactId>
<version>2.0.1</version>
1 Like
20

Hi @Ethan! The latest 3.7.3-SNAPSHOT has the change. You’ll know you’ve picked up the correct snapshot version if your log has a CoreCreatedEvent where the clientGitHash and coreGitHash properties are actual git hashes instead of a ${buildNumber} placeholder (we fixed that bug at the same time).

[com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.7.3-SNAPSHOT","clientGitHash":"aa310b0d","coreVersion":"3.7.3-SNAPSHOT","coreGitHash":"aa310b0d",...

Thanks,
David

2 Likes