I’m a big fan of online learning platforms, such as Udemy. For the price of a bottle of wine, you could learn pretty much anything, from psychology and Thai cooking to programming languages, operating systems and IT certifications. Recently, I decided to take a class with a long title: “Kubernetes Hands-On – Deploy Microservices to the AWS Cloud”. It’s a great course and if you want to learn more about nitty-gritty of Kubernetes, Microservices, cloud deployments and much more I would highly recommend it.
As a class hands-on project for deploying microservices on Kubernetes, the course author provided a small application called Fleetman. The app allows monitoring a fleet of trucks (or lorries in Brits terms) moving around the great city of Sheffield UK (though one truck is riding at the heart of London). The application generates geo-positioning data for each truck, stores it either in memory or in the database and feeds location data into a convenient map-based web UI. In the UI we could see the latest position of each truck, its speed and a current journey. An application web front-end screenshot is listed below.
Multiple trucks position reports
While internals of the application code wasn’t the primary focus of the Udemy course, (it was a Kubernetes and not a software development class anyway), I found application architecture quite fascinating, thanks to the fact that the source code is opened to the general public. As a persistence database layer, the original Fleetman application uses MongoDB, which arguably is a popular choice for storing and querying simple telemetry data coming from trucks in JSON format.
But what about Couchbase? Couchbase database platform offers a lot of features and capabilities such as N1QL language, Analytics, Eventing and Full-Text search. These features can make a simple telemetry gathering app more powerful. But how difficult would it be to re-wire the Fleetman app to use Couchbase instead of Mongo in the first place? So I spent quite some time plunging into the source code and refreshing my Java programming skills. Let’s start by looking at Fleetman’s architecture in more detail.
Fleetman architecture
Fleetman application is designed as a set of modules (or microservices) communicating with each other using various methods and protocols. Three microservices: Position Simulator, Position Tracker and Gateway API are written using Spring Boot Java application framework. The front-end module is written in Angular which is a popular Javascript development framework. There are also a messaging system and a database layer. As you might have guessed, each of these microservices would be running in a separate pod in case of Kubernetes deployment.
Fleetman architecture
- Position Simulator – generates vehicle telemetry for each of the 40 trucks in the fleet by reading configuration files with position coordinates. These position reports are written to the Message queue in JSON format, as shown below.
|
1 |
{time=2020-06-22T11:35:55.174-0400, lat=53.4002010, long=-1.4101460, vehicle=Factory Run D} |
- Position Tracker – consumes vehicle positions from the queue and stores them in Couchbase. It also calculates the speed of each truck, based on the new and previous position reports. This microservice provides a set of methods for getting the latest position of a specific truck, all trucks, as well as getting a history of all positions for a truck. These methods are available via REST interface. Getting history could be useful, if one of the K8s pods restarts.
- API Gateway – is a simple gateway for the Angular front-end. It reads latest vehicle positions from Position Tracker and feeds those into the web application.
- Web Application – is an Angular based app that allows monitoring truck positions on the map and jumping to a specific truck location while showing its traveled path. On the map below you could see the “London Riverside” truck journey.
“London Riverside” truck journey
Migration to Couchbase
Thanks to the modular approach of microservices architecture, most of my migration efforts were spent with Position Tracker module, which deals with the database layer. The original Fleetman code was written a couple of years ago, so my first goal was to recompile all the Java microservices with the newer version of Spring Boot libraries. Spring framework provides developers with fast and easy way to setup and develop regular and web applications. Spring Boot is an extension of Spring framework which comes with embedded HTTP server (Tomcat) and doesn’t require configuration via XML. It’s important to note that Couchbase is a first-class citizen as far as Spring integration goes. In June of this year the latest version 4.x of Couchbase Spring libraries have been released and full documentation is available here.
As part of the migration, one of the first things that I had to implement was adding key generation into the code in order to save documents into the database. Every document in Couchbase should have a key. Spring Boot generates the key using @GeneratedValue annotation either with the combination of document attributes or with UUID random generator. I used the later. Entity mapping for the vehicle position is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
public class VehiclePosition implements Comparable<VehiclePosition> { @NotNull @Id @GeneratedValue(strategy = UNIQUE) private String id; @NotNull @Field private String name; @NotNull @Field private BigDecimal lat; @NotNull @Field private BigDecimal longitude; @JsonFormat(shape=JsonFormat.Shape.STRING, pattern="yyyy-MM-dd'T'HH:mm:ss.SSSZ", timezone="UTC") @NotNull @Field private Date timestamp; private BigDecimal speed; …. } |
It was also necessary to add a Couchbase configuration class with a hostname, user name, password and a couple of converters to deal with storing BigDecimal values for coordinates. By default, Couchbase stores coordinates as an object with scale and precisions. It caused some issues with the application.
|
1 2 3 4 5 |
"lat": { "scale": 7, "intCompact": 533783660, "precision": 9 }, |
Using converters I was able to store truck positions data in the following format:
|
1 2 3 4 5 6 7 8 |
{ "name": "Express Delivery1", "_class": "com.virtualpairprogrammers.tracker.domain.VehiclePosition", "lat": "53.3009490", "speed": "1.333069443260374304264", "longitude": "-1.1231240", "timestamp": 1595735263364 } |
Couchbase configuration class is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
public class CustomCouchbaseConfig extends AbstractCouchbaseConfiguration { @Override protected List<String> getBootstrapHosts() { return Collections.singletonList("localhost"); } @Override protected String getBucketName() { return "test"; } @Override protected String getBucketPassword() { return "couchbase"; } @Override public CouchbaseCustomConversions customConversions() { return new CouchbaseCustomConversions(Arrays.asList(BigDecimalToString.INSTANCE, StringToBigDecimalConverter.INSTANCE)); } @WritingConverter public enum BigDecimalToString implements Converter<BigDecimal, String> { INSTANCE; @Override public String convert(BigDecimal source) { return source.toString() ; } } @ReadingConverter public enum StringToBigDecimalConverter implements Converter<String, BigDecimal> { INSTANCE; @Override public BigDecimal convert(String source) { return new BigDecimal(source); } } } |
To configure Couchbase I created a test bucket, added an RBAC user with the same name as the bucket and granted Application access permissions to this user. I also added indexes on name and timestamp since these two attributes are used as access patterns by repository methods. PositionRepository interface has three methods to search trucks by name, timestamp and a combination of both.
|
1 2 3 4 5 6 7 8 9 |
public interface PositionRepository extends CouchbasePagingAndSortingRepository<VehiclePosition, String> { TreeSet<VehiclePosition> findByNameAndTimestampAfterOrderByTimestampAsc(String name, Date timestamp); List<VehiclePosition> findByNameOrderByTimestampAsc(String name); List<VehiclePosition> findByTimestampAfter(Date since); } |
Summary
Overall, migration to Couchbase went pretty smoothly. I tested all the code locally but it can be containerized and deployed on K8s in the cloud using Couchbase Autonomous Operator for Kubernetes. Such an example is presented here. From the enterprise perspective, using other Couchbase features we could potentially extend Fleetman app capabilities. For instance, using N1QL or Analytics developers might construct a query that would find the top 5 trucks based on the max distance traveled over a certain timeframe. Eventing feature could be used to send alerts if a certain truck exceeds a speed limit. Another opportunity for improvement would be to use Couchbase Geospatial queries. The source code of my version of Fleetman application is located on GitHub right here.
