I Bought an Expensive US Market Dataset This Year, Then Found This Public One

Nothing is clearer than what has been tested; no argument is settled without evidence.
—— Wang Chong, Lunheng (Eastern Han, c. 1st century; translation mine)

If you want to do your own US market research, the first wall you hit is usually data.

Public sources tend to give you a few recent years, only companies still trading, or a rate limit that stops you after a few dozen queries. Anything complete starts at several hundred dollars a year.

I paid that this year. I bought Norgate at $40 a month, close to five hundred dollars a year. It is a well-regarded US market history database, and I felt good about the money: clean data, delisted names included, maintained by people who care. That is what research needs.

Last week, while looking for something else, I came across a public dataset. I opened it and stopped.

Much of what it holds overlaps with what I paid for.

What is in it

The link: defeatbeta/yahoo-finance-data

Download the whole thing, no account, no rate limit, refreshed daily.

Here is what I found when I queried it (as-of 2026-09-04):

• Daily prices: 36.5 million rows, 12,227 tickers, November 1994 to today
• Earnings call transcripts: 238,638 calls, 6,511 tickers, 2005 to today
• Financial statements, full-text company news, an SEC filing index, revenue breakdowns, dividend and split records

The set I pay for covers 12,259 tickers. This one covers 12,227. I read those two numbers twice.

Two things that surprised me

The first is that delisted companies are still in there.

This matters more than it sounds. Many datasets only carry companies still trading today, dropping everything that went bankrupt, got acquired, or was delisted. Compute performance from the survivors and the numbers look great, because the losers are not in the sample. It is called survivorship bias, and it never throws an error — your code runs fine and your conclusion is skewed.

I pulled 12 tickers that stopped trading years ago. All 12 were there, with history preserved through their final day. AMED stops on 15 August 2025. MDRX stops on 20 February 2026.

The second is that the transcripts are already split by speaker.

Not one block of text you have to segment yourself, but every passage tagged with its position and who was talking. You can query which company said a given word in which quarter, or read only the Q&A section — the question analysts press hardest on is usually where the market is least comfortable.

One thing to know about

Prices are adjusted for splits, not for dividends.

The split part it gets right. A company does a four-for-one split, one share becomes four, and the price goes from four hundred to one hundred overnight. If the data ignores that, your program reads a 75% one-day drop. I checked Apple and Nvidia on their split dates; both line up.

Dividends are not handled. That means computing long-run returns straight from it will understate them, and dividend payers understate most. It is recoverable, but you have to know it is there first.

I would not call this a flaw. It is an assumption. Paid data ships with documentation telling you how each column was computed; public data usually ships as a file. The file will not lie to you, but it will not volunteer what it left out either.

How to start

The files are parquet, so a browser cannot open them and you need code. The upside is you can query without downloading, straight against the URL:

import duckdb
c = duckdb.connect()
c.execute("INSTALL httpfs; LOAD httpfs;")
u = "https://huggingface.co/datasets/defeatbeta/yahoo-finance-data/resolve/main/data/stock_prices.parquet"
print(c.execute(f"SELECT * FROM read_parquet('{u}') WHERE symbol='AAPL' LIMIT 5").fetchall())

Pulling the whole thing local works too. Sizes for reference: prices 462 MB, transcripts 2.26 GB, news 1.1 GB, everything else a few MB.

There is also a ready-made package: pip install defeatbeta-api.

Three questions for any new dataset

This exercise left me with three questions I now ask of anything new.

First, are the losers still in there? Does this data keep the failed cases? If it only keeps survivors, every average you compute runs high. Not a markets-only problem: fitness program results, startup success rates, the track record of any investing personality — same question.

Second, does it agree with something I already know? Find one example where you know the answer and look it up. Apple the day before its split, an index close from last year, anything. If it disagrees, stop before you build on it.

Third, and this is the one I nearly missed: can my check tell the difference?

The first time I tested dividend handling, the tickers I sampled paid no dividends, so every check came back clean. That clean result was not the data being sound. It was my ruler being unable to measure. Swapping in dividend payers is what made the difference visible.

A ruler that never fails measures nothing.

One Thing to Take Away

When a check comes back clean, ask one more question: if something were wrong, would this check have caught it?

If not, that clean result is not good news. It is no news.

A practice for today. Think of something you recently confirmed was fine — a normal health screening, a project that seems on schedule, a kid who says school is going well. Pick one, then ask: how exactly did I confirm that? If it were going badly, would my method have shown me?

Most of the time the answer is yes. But there will be one or two where you notice you only ever asked a question that was always going to come back clean.

Raw Markdown