The Cluster
at a glance
Sources → ingestion → storage → Spark compute → research → observability. See how the pieces stay deterministic and debuggable.
I design and operate a trading data platform
A Problem Clearly Stated is a Problem Half Solved
Tackling big data challenges with modern tools
How It Started
A Multi-Year "Weekend Project"
In 2021 I became curious about the stock market. Over a weekend I hacked together a small Ruby on Rails app, scraped historical prices from Yahoo! Finance, and stored them in a local postgres database. I started with a simple question: which S&P 500 stocks tend to move together, and which tend to move in opposite directions? I ran a simple analysis comparing returns across a universe of 50 tickers. It took 3 days to finish.
The results were interesting, but too slow to be useful. I wanted to refresh the analysis daily and include the entire market. Scaling to 10,000 tickers would amount to a naïve work factor increase of over 120,000×. My single threaded Rails job running on a spare laptop wasn't up to the task.
How It's Going
I 🧡 Data Engineering!
Since then I've built out a research and trading platform, from bare metal hardware to streaming Spark pipelines powering live trading algorithms. Modern OSS and consumer hardware make it possible to build powerful systems.
- 6 nodes, 92 cores/184 threads, 750GB RAM, 48TB HDFS storage
- Batch jobs coordinated via Airflow
- Live data from brokers straight into Kafka - 100 tickers, thousands of trades per second
- Spark 4.1 Structured Streaming into Delta Lake
- 25 years of historical 1-min data, updated daily
- 2+ TB of parquet/Delta tables
The Right Tool for the Job
When All You Have is a Hammer…
I started on a single laptop, scraping public data and running a single-threaded Rails job. It was the framework I knew best from 10+ years of full stack web development. Unfortunately, it wasn't going to help me work with market data at scale.
I love Rails - it changed my life. The ORM and MVC architecture made sense to me and I was able to use it to solve the problems for myself and others. I had the same experience a second time when I spent a year learning Flutter. Suddenly mobile development was in my skill set.
Fortunately, lightning struck a third time for me. Learning Spark and having the power to solve problems I thought were out of my league was thrilling. Big data engineering is in my toolbelt now, and I am excited to see where it takes me next.
Employment
I'm Looking for a full‑time data engineering role
I've built this platform alongside full‑time software engineering roles, delivering features and supporting production systems while deepening my data engineering skills.
I enjoy partnering with data scientists and other engineers to stabilize and optimize existing stacks. In my next role, I’m looking to focus that experience on data engineering problems that matter to your business.