From Stream To Database: Processing Market Data With Spring Boot, Redis, And Flyway
Hello everyone!
In my last post, we saw how our Python service collects B3 data and publishes it to Kafka. Today, we take a crucial step: consuming this data and making it useful for our brokerage ecosystem.
I’ll introduce the Broker Asset API, the Java microservice responsible for managing the asset catalog, keeping prices updated, and serving this information with ultra-low latency.
???? MVP Focus (Minimum Viable Product)
Before diving into the code, a quick disclaimer: we are building the foundation. At this stage, the goal is to ensure the end-to-end flow works seamlessly.
The focus is on delivering core value: making data available and performant. In the future, we will revisit this service to add unit tests, refine exception handling, and increase resilience.
????️ The Pillars of the Asset API
For this MVP, I focused on four main implementation points:
1. Database Evolution with Flyway
To ensure our MySQL schema is versioned and reproducible, I used Flyway. We created the assets table to store the ticker, name, current price, and asset status.
-
Technical Highlight: Using an index on the
tickerfield ensures that symbol-based queries are extremely fast.
2. Kafka Consumer: Real-Time Reactivity
The API "listens" to the trading-assets-market-data-v1 topic. As soon as the Python service publishes a new price, our consumer captures the message and triggers the update flow.
- Ordering Guarantee: Since we use the ticker as the Kafka key, we process updates in the exact order they were generated.
@KafkaListener(
topics = "trading-assets-market-data-v1",
groupId = "broker-asset-api"
)
public void consume(AssetMarketDataDTO dto) {
log.info("Received market data for: {}", dto.getTicker());
assetService.updateAsset(dto);
}
3. Service Layer: Hybrid Persistence (SQL + Redis)
The AssetService is where the business logic resides. Upon receiving a new price, it performs two operations:
- MySQL Persistence: Updates the asset record (or creates a new one) to ensure data consistency.
-
Cache Update (Redis): The price is sent to a Redis cache with the key
market:price:{ticker}. - This allows other system components to query prices instantly without overloading the relational database.
4. Controller: Serving the Information
We created REST endpoints so that other services or the front-end can query the asset catalog:
-
GET /api/v1/assets: Lists all available active assets. -
GET /api/v1/assets/{ticker}: Returns details and the updated price for a specific asset.
????️ What’s Next?
The Asset microservice is the foundation that ensures the system knows "what" is being traded and "for how much." Since we are following an MVP strategy, the focus was on establishing this data contract and basic persistence.
But an asset alone doesn't make a brokerage. It needs to belong to someone.
In the next post, we’ll increase the complexity and talk about the Broker Wallet API. We will explore:
- Managing user financial balances.
- Linking asset custody to customer wallets.
- How the system reflects market variations in the investor's equity.
What do you think of this step-by-step approach to building an ecosystem? Let me know your thoughts in the comments!
???? About the series
⬅️ Previous Post: Tooling Tips.
???? Series Index: Series Guide.
Links:
Popular Products
-
Classic Oversized Teddy Bear$23.78 -
Gem's Ballet Natural Garnet Gemstone ...$171.56$85.78 -
Butt Lifting Body Shaper Shorts$95.56$47.78 -
Slimming Waist Trainer & Thigh Trimmer$67.56$33.78 -
Realistic Fake Poop Prank Toys$99.56$49.78