Previously, we shared a post on how to monitor Stellar assets using Couchbase. In this follow-up, we’ll demonstrate how to load recent Stellar blockchain transactions (e.g., trades) into our Capella DBaaS, making them easily accessible for use in sample applications. With Capella’s built-in web management UI, you can run test queries, explore documents, and perform various other tasks to interact with the blockchain data seamlessly.
What are Stellar trades?
Stellar is a decentralized blockchain platform designed for fast, secure, and low-cost transactions. For developers and database backend engineers, Stellar offers a unique opportunity to build financial applications that can handle cross-border payments, micropayments, and tokenized assets at scale. What sets Stellar apart is its consensus algorithm, the Stellar Consensus Protocol (SCP), which provides a more energy-efficient alternative to traditional proof-of-work blockchains. This allows developers to integrate blockchain functionality without compromising on speed or scalability.
Stellar’s API and SDKs support a wide range of programming languages, making it easier to build, connect, and manage blockchain-backed applications within existing database infrastructures. By leveraging Stellar, developers can bring the power of decentralized finance (DeFi) into their systems with minimal friction and maximum impact.
Stellar trades are operations that occur on the Stellar blockchain, facilitating the transfer of assets between accounts or interacting with the network in various ways. Each transaction is made up of a set of operations and requires authorization from the initiating account holder via a digital signature. Stellar supports several types of operations within a single transaction, making it highly versatile.
Key aspects of Stellar transactions include:
-
- Asset Transfer: Stellar allows the transfer of any type of asset, including native XLM (Lumens) and tokenized fiat currencies, between accounts.
- Path Payments: This feature enables users to send one type of asset while the recipient receives another. Stellar’s decentralized exchange (DEX) automatically finds the best exchange path to facilitate the conversion.
- Creating and Managing Accounts: Transactions can create new accounts, assign starting balances, and set various account parameters like trustlines (to accept new asset types).
- Smart Contracts (Multi-Sig): While Stellar does not have traditional smart contracts like Ethereum, it supports multi-signature transactions and time-locked operations, allowing more complex transaction scenarios such as escrow-like setups.
- Issuing and Trading Assets: Stellar enables anyone to issue custom assets (such as tokens) and trade them on its DEX. Developers can tokenize assets and provide liquidity for trading pairs.
- Fee Structure: Stellar transactions are designed to be fast and low-cost, with fees being just fractions of a cent (denominated in XLM). This is particularly beneficial for applications requiring high transaction throughput and minimal cost overheads.
For developers, Stellar transactions offer an easy way to interact with digital assets and integrate decentralized functionality into applications while benefiting from the blockchain’s fast confirmation times and low fees.
How to Load Historical Stellar Data from Horizon Server Using Python
The Stellar Horizon service plays a vital role in the Stellar ecosystem, serving as the bridge between developers and the Stellar network. By providing a RESTful API, it allows applications to easily interact with the Stellar blockchain without needing to directly access the underlying Stellar Core nodes. This abstraction simplifies the process for developers, making it more straightforward to build and integrate blockchain-powered features.
Horizon handles several key functions essential for blockchain interaction. It enables developers to submit transactions, validate them, and forward them to Stellar Core for processing. Additionally, it offers access to detailed account information, such as balances, transaction history, and trustlines, which simplifies data retrieval and display. Horizon also allows developers to query the Stellar ledger for past transactions and operational details, serving as a user-friendly gateway to access both real-time and historical blockchain data.
In this example we are only going to look at trades and pricing of assets based on historical trades. The trades documents, returned from Horizon as JSON, look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
{ "_links": { "self": { "href": "" }, "base": { "href": "https://horizon.stellar.org/liquidity_pools/94b45fd827c6d0a420032d600e8738625efd0ee7c6e4d6345cb184ee6e153ebb" }, "counter": { "href": "https://horizon.stellar.org/accounts/GD65FION32RSQYIHDSQU3ZNMHWCFQ4KVILQYDFGFWI3NCCZXQG265EXP" }, "operation": { "href": "https://horizon.stellar.org/operations/202488814445412353" } }, "id": "202488814445412353-0", "paging_token": "202488814445412353-0", "ledger_close_time": "2023-07-11T02:54:21Z", "trade_type": "liquidity_pool", "liquidity_pool_fee_bp": 30, "base_liquidity_pool_id": "94b45fd827c6d0a420032d600e8738625efd0ee7c6e4d6345cb184ee6e153ebb", "base_amount": "0.0000003", "base_asset_type": "native", "counter_offer_id": "4814174832872800257", "counter_account": "GD65FION32RSQYIHDSQU3ZNMHWCFQ4KVILQYDFGFWI3NCCZXQG265EXP", "counter_amount": "11.0000000", "counter_asset_type": "credit_alphanum4", "counter_asset_code": "wSHX", "counter_asset_issuer": "GDAPJ5EVAXUU5ZLDHBUQF3MA5EV7S44T3AJPY42LMSD6ZZKQOQIAWSHX", "base_is_seller": true, "price": { "n": "110000000", "d": "3" } } |
Our Python script will download the most recent trades and upload the documents into Capella.
Requirements for connecting to Horizon and Couchbase
Couchbase and Stellar both have a couple requirements before we can run our script:
-
- We use the Python stellar_sdk module. In previous examples we used basic HTTP requests/fetches instead, but this way we can keep the code simple and use the supported libraries instead. Install with
pip install stellar-sdk
. - If you want to only look at trades relating to certain assets, you will need to know the asset identifiers beforehand. The asset_code identifies the specific asset being traded, while the asset_issuer specifies the Stellar account that issued the asset, ensuring its authenticity and origin.
- Beware: both elements are critical to identify the correct asset when trading!
- For connecting to Capella, you must have an account – sign up for a free Capella account today, with no obligations.
- Your cluster must have a bucket named stellar.
- You must also know the connection URL to the bucket you are working with.
- Cluster access must be enabled with a name and password as well.
- These settings are accessible through the Capella management interface.
- Save these settings in a local_settings.py file that we will call with our script:
1234endpoint = "couchbases://cb.xlzvmx....jom.cloud.couchbase.com"username = "clusterusername"password = "clusterpassword"bucket = "stellar"
- We use the Python stellar_sdk module. In previous examples we used basic HTTP requests/fetches instead, but this way we can keep the code simple and use the supported libraries instead. Install with
Accessing Stellar Horizon data with Python
With the above in place, here is a sample of the basic code function to grab trades, optionally filtering by asset_code or asset_issuer:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
def fetch_and_store_trades(asset_code=None, asset_issuer=None): server = Server(horizon_url=HORIZON_URL) trades_call_builder = server.trades().limit(MAX_TRANSACTIONS) try: trades_response = trades_call_builder.call() trades = trades_response['_embedded']['records'] # Filter trades manually based on asset code and issuer if asset_code and asset_issuer: filtered_trades = [ trade for trade in trades if ( (trade.get('base_asset_code') == asset_code and trade.get('base_asset_issuer') == asset_issuer) or (trade.get('counter_asset_code') == asset_code and trade.get('counter_asset_issuer') == asset_issuer) ) ] else: filtered_trades = trades # Compute price as a floating-point number for trade in filtered_trades: price_n = float(trade['price']['n']) price_d = float(trade['price']['d']) trade['price_number'] = price_n / price_d upload_to_couchbase(filtered_trades, COUCHBASE_BUCKET) except Exception as e: print(f"An error occurred while fetching trades: {e}") |
Decoding the Stellar asset trade price
For this example we are specifically interested in three sets of information shown in the earlier sample document:
-
- asset_code & asset_issuer – lines 28-29 in sample document
- amounted traded – line 26
- trade prices – lines 31-34 these tell us what the price of the asset was during the trade and are held in a JSON object called price:{}
The first two sets of data are straightforward attributes we can use as-is. But prices are a little more complicated. Prices on Stellar have very high precision and do not just use floats. Instead they use a numerator and denominator that are divided, as needed, to produce a floating point price. So, as shown in the code sample, on line 25, we compute a new attribute to be saved in the document before it is uploaded to the database.
trade['price_number'] = price_n / price_d
You could use the same pattern to add any kind of attribute you need, like download date, project name, etc.
Now that we have the trade document, added our normal looking price number, we can upload it to the database with our upload function code.
Uploading documents to Capella with Python
There are a few approaches to uploading and/or updating documents in Capella. In this case we use the upsert function to add or replace documents with the same ID. We take the ID straight from the trade’s ID attribute, so our flexible custom key for our documents can reflect Horizon exactly. This is helpful for future alignment with historical data from Horizon as you scale your system.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
def upload_to_couchbase(data, bucket_name): try: # Connect to Couchbase cluster timeout_options = ClusterTimeoutOptions(kv_timeout=timedelta(seconds=10)) cluster = Cluster( COUCHBASE_CLUSTER, ClusterOptions(PasswordAuthenticator(COUCHBASE_USERNAME, COUCHBASE_PASSWORD), timeout_options=timeout_options) ) bucket = cluster.bucket(bucket_name) collection = bucket.default_collection() # Upsert the data (overwrite if key exists) for trade in data: key = trade['id'] collection.upsert(key, trade) print(f"Data has been uploaded to Couchbase bucket '{bucket_name}'") except CouchbaseException as e: print(f"An error occurred while uploading to Couchbase: {e}") |
The full script is included here, click to expand:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
import argparse import requests import json from datetime import timedelta from couchbase.cluster import Cluster from couchbase.auth import PasswordAuthenticator from couchbase.options import ClusterOptions, ClusterTimeoutOptions from couchbase.exceptions import CouchbaseException from stellar_sdk.server import Server import local_settings as capella # Define the Horizon API endpoint HORIZON_URL = "https://horizon.stellar.org" MAX_TRANSACTIONS = 100 # Number of transactions to retrieve BATCH_SIZE = 200 # Number of transactions per request (maximum allowed by the API) # Couchbase connection details COUCHBASE_CLUSTER = capella.endpoint COUCHBASE_BUCKET = capella.bucket COUCHBASE_USERNAME = capella.username COUCHBASE_PASSWORD = capella.password def fetch_and_store_trades(asset_code=None, asset_issuer=None): server = Server(horizon_url=HORIZON_URL) trades_call_builder = server.trades().limit(MAX_TRANSACTIONS) try: trades_response = trades_call_builder.call() trades = trades_response['_embedded']['records'] # Filter trades manually based on asset code and issuer if asset_code and asset_issuer: filtered_trades = [ trade for trade in trades if ( (trade.get('base_asset_code') == asset_code and trade.get('base_asset_issuer') == asset_issuer) or (trade.get('counter_asset_code') == asset_code and trade.get('counter_asset_issuer') == asset_issuer) ) ] else: filtered_trades = trades # Compute price as a floating-point number for trade in filtered_trades: price_n = float(trade['price']['n']) price_d = float(trade['price']['d']) trade['price_number'] = price_n / price_d upload_to_couchbase(filtered_trades, COUCHBASE_BUCKET) except Exception as e: print(f"An error occurred while fetching trades: {e}") def upload_to_couchbase(data, bucket_name): try: # Connect to Couchbase cluster timeout_options = ClusterTimeoutOptions(kv_timeout=timedelta(seconds=10)) cluster = Cluster( COUCHBASE_CLUSTER, ClusterOptions(PasswordAuthenticator(COUCHBASE_USERNAME, COUCHBASE_PASSWORD), timeout_options=timeout_options) ) bucket = cluster.bucket(bucket_name) collection = bucket.default_collection() # Upsert the data (overwrite if key exists) for trade in data: key = trade['id'] collection.upsert(key, trade) print(f"Data has been uploaded to Couchbase bucket '{bucket_name}'") except CouchbaseException as e: print(f"An error occurred while uploading to Couchbase: {e}") def main(): parser = argparse.ArgumentParser(description="Fetch trades from Stellar and store in Couchbase.") parser.add_argument('--asset_code', type=str, help='The asset code to filter trades by.') parser.add_argument('--asset_issuer', type=str, help='The asset issuer to filter trades by.') args = parser.parse_args() fetch_and_store_trades(args.asset_code, args.asset_issuer) if __name__ == "__main__": main() |
Viewing documents in Capella
Go to the Data Tools tab in Capella and set the Bucket drop-down to stellar and use _default for the scope and collection.
The documents are listed and you can click on the blue numeric DOC ID to see the full document and then scroll down to see the pricing info:
Querying Stellar blockchain trades in Capella
Now that the data is in Capella we can index it and query it. But let’s try it in a slightly non-conventional way by running our query first. Capella is smart and can suggest what indexes need to be made. So when you run a query and it notices you could optimize them further, it will make suggestions. You accept the suggestion and can then carry on.
Go to the Data Tools -> Query tab and select the Stellar bucket from the drop-down above the query window.
Erase the default query and enter this one to get a list of trades with their IDs, price, and the price that Python calculated, along with on-the-fly pricing calculations within the Capella SQL++ query language:
1 2 3 4 5 6 |
SELECT id, price, price.n AS price_number, TONUMBER(price.n) / TONUMBER(price.d) AS price_calc FROM stellar._default._default |
As you can see, this basic SQL, though you can see in the FROM statement that we could have scope and collections and subsets of documents if we wanted to.
The results are listed in JSON by default, below, or you can choose to see them in a table. Capella is powerful because it can show you JSON objects within documents and not only basic field types typically used in SQL databases, note the price object.
Create indexes for trades
Capella automatically provides indexing advice in the top right corner. There is also a tab on the middle right of the screen in case it is hiding on you.
Choose Build Suggested and your queries will run much faster – mine went from 600ms to 6ms after indexing.
This video shows similar interaction in a live view, so you can see it in action, though it uses a different dataset:
What next?
Keep on building your apps using Capella’s free version by simply signing up or pick up a starter pack and also get some serious discounts on cloud credits and powerful support from the Couchbase team.