Scott Williams - Personal Website

About Me

Hi, I'm Scott, a software engineer and builder with a focus on AI/ML, cloud infrastructure, and automated systems. My recent independent work spans self-hosted operations, full-stack development, and GCP-based data pipelines applied to projects ranging from social media automation to personal finance research.

I learn and operate best by building. Recent projects include a data pipeline for automating arbitrage trades on Kalshi's prediction markets using order book and weather data, an AI-based video appraisal system, an MCP server for LLM debugging, tooling for rapidly deploying VPS-based full-stack applications, and LangGraph-based AI agents and workflows. These projects have deepened my expertise in cloud architecture, AI engineering, infrastructure as code, and end-to-end system design.

From June 2025 through January 2026, I worked as a Data Engineer at iCapital in New York City on the Data & Analytics team. I played a pivotal role in migrating mission-critical integrations from Airflow to Prefect and architected a Kafka-based streaming ingestion solution to bring real-time data from the platform into our data infrastructure.

I hold a Bachelor's in Computer Science with a focus in Applied Mathematics from the Colorado School of Mines. My academic background gave me a strong foundation in CS theory and its mathematical underpinnings, knowledge that continues to inform my work in the AI/ML space.

Outside of work, I'm a lifelong athlete. I've played hockey since age six and captained the rowing team at The Taft School. I also enjoy poker, hiking, running, and keeping up with AI research.

Thanks for visiting. Feel free to connect through the links above.

Work Experience

iCapital, Data Engineering Analyst (June 2025 - January 2026):
Architected a Kafka-based streaming ingestion solution to bring real-time data from the platform into our data infrastructure. Played a pivotal role in migrating mission-critical integrations from Airflow to Prefect. Designed document parsing pipelines to assist business stakeholders with repetitive document searching and information retrieval processes.
iCapital, Data Engineering Intern (Summer 2024):
Researched Snowflake usage attribution and built DBT models supporting Tableau dashboards.
Colorado School of Mines Career Center, Student Assistant (2022-2024):
Coordinated Career Day and resolved technical issues for students and faculty.
Coolipy, Open Source Contributor (February 2025 - Present):
Collaborated with maintainers to identify and resolve bugs, contributing merged fixes to the Python client library for Coolify.

Awards

Google Gemini NYC Hackathon 3rd Place ($20k Gemini Credits) (November 2025):
Built an LLM debugging layer for coding agents. BugPoint is an LLM-native debugging system that provides AI coding agents with structured visibility into code execution through control-flow graphs and runtime traces, enabling agents to reason about complex control flow within tight context windows.
Demo, GitHub Repo.
Meta Llama 4 NYC Hackathon 2nd Place ($5k Prize) (June 2025):
Competed in a 24 hour contest against hundreds of talented developers to build AthletIQ, a personalized AI coach which gives athletes feedback based on video footage from any sport. Achieved 2nd place overall in the contest.
Demo Video, Hackathon Presentation, LinkedIn Post, GitHub Repo.

Projects

SaaS Factory: Developed a CLI tool that automates VPS provisioning and configuration for rapid SaaS deployment. The tool spins up Linode instances (with plans to extend to additional providers like Hostinger and AWS EC2), automatically installs Coolify for VPS management, and configures open source products including N8N, Supabase, and Prefect. Additionally supports deployment of fully hosted Next.js starter applications with GitHub webhook integration for seamless CI/CD workflows, enabling developers to launch production-ready SaaS products in minutes with less than 5 commands and limited UI interaction. Currently documenting the tool to make it more user-friendly and accessible to developers of all experience levels.
CaveatBot: Built a VS code extension in under 36 hours for a hackathon. The extension records all code changes, commands and their output, screenshots, and notes and adds them to a knowledgebase, allowing fast and efficient querying to recall configurations made to a project using Groq for fast inference and Pinecone's vector DB.
Model Fine-Tuning & Synthetic Data Generation Micro-Project : Created a pipeline to filter and label reddit content for finetuning jobs using small open source models. This was ultimately used to fine tune the 1 billion parameter gemma 3 model to run locally for easy json extraction. Link to model: Hugging Face Model and link to dataset: Hugging Face Dataset.
News Video Generator: This project automates the creation of short-form videos for TikTok and Instagram based on a topic prompt. It leverages web scraping to source recent news articles, utilizes semantic search and GPT-4-powered script generation, incorporates a custom text-to-speech API for narration, automates image sourcing, and integrates dynamic "white noise" video. Using FFMPEG for fast and automated video processing, the gathered assets are seamlessly organized and rendered into engaging, high-retention content optimized for social media platforms. See sample posts on Instagram: World News Daily & Tech News Daily.
Personal Portfolio Builder: Created a chatbot to quickly generate and edit professional-quality portfolio websites.
Clue Game Project: Recreated the game of Clue using Java with Test Driven Development and collaborative coding.

Skills

Programming Languages:

C++, Java, Python, OCaml, JavaScript, Jinja2, R, TypeScript

Web Development Frameworks and Tools:

Angular, Next.js, React, Vite, FastAPI, Auth.js, Lucia Auth, Supabase, PocketBase, Cloudflare

Data Management:

PostgreSQL, MySQL, Snowflake, DBT, Kafka, Vector Databases, Airflow, MongoDB

Cloud and Virtualization Technologies:

AWS (Lambda, EventBridge, MWAA, EC2, ECR, RDS), Google Cloud, Docker, Linode (VPS), Coolify (VPS management software)

Networking and Proxy Management:

DNS, Traefik

AI Engineering and Large Language Models (LLMs):

LangChain, LangGraph, Tavily, Vector Databases, Model Fine-Tuning, Model Quantization, Synthetic Data Generation

Machine Learning and Statistical Modeling:

TensorFlow, Transformers, Scikit-Learn (SKLearn), SciPy, OpenCV, Time Series Analysis, Linear/Polynomial Forecasting, Stochastic Modeling (Markov Chains, Poisson Processes, Brownian Motion)

Video Creation and Editing:

FFMPEG, GarageBand, Adobe Premiere Pro, Clipchamp

Project Management:

Agile, Test-Driven Development (TDD), Design Documentation and Review, Sprint Planning

Terminal and Tools:

Bash, PowerShell, Git, AWS CDK

Education