Massive publishes bulk historical market data as flat files, reachable with any S3-compatible client. You need three things: a paid subscription that includes flat files, the S3 Access and Secret keys from your dashboard, and the endpoint. Point the client at https://files.massive.com with the bucket flatfiles and everything else works the way S3 normally does.
Applies to
- Plans: paid plans that include flat files. The free Basic tier has no flat-file access.
- Endpoints: https://files.massive.com, bucket flatfiles. This is separate from the REST API and uses different credentials from your API key.
- Asset classes: stocks, options, indices, futures, forex and crypto, each under its own prefix.
What you need first
What you need first is a subscription, a pair of S3 keys and, for real-time data, a signature:
- A paid subscription including flat files. Check the pricing page for which plans include them.
- Your S3 Access Key and Secret Key from the dashboard. These are not your REST API key and cannot be used in its place.
- If your plan includes real-time data, sign the exchange agreements on the agreements page. An unsigned agreement produces a 403 that looks like a credentials problem and is not one.
How the files are organized
The files are organized by asset class, dataset and date, one file per trading day, gzipped CSV:
flatfiles/us_stocks_sip/trades_v1/2024/03/2024-03-07.csv.gz
^ prefix ^ dataset ^ year/month/day
The prefixes for stocks, options, indices, forex and crypto are us_stocks_sip, us_options_opra, us_indices, global_forex and global_crypto. Futures are arranged differently: rather than one asset-class prefix they are split by exchange, with CBOT, CME, COMEX and NYMEX each carrying their own minute aggregates and quotes. List the bucket root to see exactly which prefixes and datasets your plan can reach rather than guessing at paths.
What is inside a file
A stocks trades file is gzipped CSV with a header line and one row per trade. These are its thirteen columns, in file order, from the flat-file trades documentation:
| Column | Holds |
|---|---|
| ticker | The symbol the trade executed under |
| conditions | Sale condition codes. Several are quoted and comma-separated, as in "12,37" |
| correction | Correction indicator; empty on a normal trade |
| exchange | Exchange id, as returned by the exchanges endpoint |
| id | Trade id, unique per combination of ticker, exchange and TRF |
| participant_timestamp | When the exchange generated the trade |
| price | Dollar value per whole share |
| sequence_number | Order of events for this ticker; increasing, not contiguous, reset daily |
| sip_timestamp | When the SIP received the trade from the exchange |
| size | Number of shares |
| tape | 1 is NYSE-listed, 2 is NYSE Arca and American, 3 is Nasdaq-listed |
| trf_id | Trade Reporting Facility id; empty on an exchange print |
| trf_timestamp | When the TRF received the trade |
All three timestamps are nanoseconds since the Unix epoch in UTC. The raw rows are long, so here are two of them as the file actually stores them:
ticker,conditions,correction,exchange,id,participant_timestamp,price,sequence_number,sip_timestamp,size,tape,trf_id,trf_timestamp
MSFT,"12,37",,11,1,1679990400018025984,276.16,1550,1679990400018381329,55,3,,0
MSFT,12,,8,1,1679990400000110000,276.75,1556,1679990400034839796,100,3,,0
How to connect with each client
Four clients are tested and supported. Pick one; the credentials and endpoint are the same for all of them.
AWS CLI
aws configure set aws_access_key_id YOUR-ACCESS-KEY
aws configure set aws_secret_access_key YOUR-SECRET-KEY
aws s3 ls s3://flatfiles/us_stocks_sip/ --endpoint-url https://files.massive.com
aws s3 cp s3://flatfiles/us_stocks_sip/trades_v1/2024/03/2024-03-07.csv.gz . --endpoint-url https://files.massive.com
Rclone
rclone config create s3massive s3 env_auth=false access_key_id=YOUR-ACCESS-KEY secret_access_key=YOUR-SECRET-KEY endpoint=https://files.massive.com
rclone ls s3massive:flatfiles/us_stocks_sip/
rclone copy s3massive:flatfiles/us_stocks_sip/trades_v1/2024/03/2024-03-07.csv.gz .
MinIO client
mc alias set s3massive https://files.massive.com YOUR-ACCESS-KEY YOUR-SECRET-KEY
mc ls s3massive/flatfiles/us_stocks_sip/
mc cat s3massive/flatfiles/us_stocks_sip/trades_v1/2024/03/2024-03-07.csv.gz | gzcat | head -4
Python, with boto3
import boto3
from botocore.config import Config
s3 = boto3.Session(
aws_access_key_id="YOUR-ACCESS-KEY",
aws_secret_access_key="YOUR-SECRET-KEY",
).client("s3", endpoint_url="https://files.massive.com",
config=Config(signature_version="s3v4"))
# List what you can reach under one prefix
for page in s3.get_paginator("list_objects_v2").paginate(
Bucket="flatfiles", Prefix="us_stocks_sip/trades_v1/2024/03/"):
for obj in page.get("Contents", []):
print(obj["Key"])
s3.download_file("flatfiles",
"us_stocks_sip/trades_v1/2024/03/2024-03-07.csv.gz",
"2024-03-07.csv.gz")
Paginate rather than listing a whole prefix in one call. A single asset class holds years of daily files.
If you would rather not configure a client
If you would rather not configure a client, the File Browser in the flat files documentation lets you browse and download from the page. It is also the quickest way to find out whether your plan can reach a given file, which makes it the right first stop when you are debugging a 403.
Why flat files rather than the REST API
Flat files exist because paging years of tick data through REST means hundreds of millions of rows and a request for every page. One daily file holds every trade for that session, so a backfill becomes a download loop rather than a paging loop, and it does not count against anything. Use REST for a window you can name and flat files for history you intend to keep.
If you see an error
Four things produce a 403 Forbidden on a flat-file download, and the response does not say which:
- Your plan does not include that dataset.
- The file is older than your plan's history. Each flat-file documentation page lists the history per plan; the stock day aggregates, for example, go back 5 years on Stocks Starter, 10 years on Stocks Developer and to September 10, 2003 on Stocks Advanced.
- The file has not been published yet. A trading day's files are available by about 11:00 AM ET the following day, according to the flat files quickstart. A download of the current day's file, which is not published until after the session, is refused with a 403 rather than a 404.
- The exchange agreements are unsigned.
Downloading through the file browser in the docs returns the same refusal as {"status":"NOT_AUTHORIZED","message":"forbidden"}, and it is no easier to read. Browsing the File Browser still answers the plan and history questions directly, so check it first. Then list the parent prefix: a file that has not been published yet is not in the listing. Then check the agreements page.
If the agreements page shows nothing left to sign and the 403 persists, look for an Update Status button at the top right of the agreements page. Choose it and confirm your subscriber details again. The dashboard can also show a notice that your data is delayed, with a link to complete your agreements; follow that link if you see it.
SignatureDoesNotMatch means the client is signing with the wrong algorithm or the secret is wrong. Confirm signature_version is s3v4 and re-copy the secret.
A 404 on an object key that looks right is usually a date with no trading, or the wrong dataset version. List the parent prefix and read the real key.
Using your REST API key as the S3 Access Key will fail. They are separate credentials issued on the same dashboard page.
