Direct Indexing in a jupyter notebook

My brother-in-law sent me a text asking me to explain how I can own SPY without owning TSLA. I told him it was too complicated to explain over text but promised a response "soon". This post is that response.

Firstly, this is not investment or tax advice. I am not a financial advisor or a tax preparer. I am not responsible for any losses you may incur or if the IRS comes knocking at your door.

Secondly, this approach will have some tracking error! In this example, I am assuming you can't buy factional shares (that's probably a bad idea) and I'm sure there are like a thousand other reasons I don't even know about. If you want to follow the S&P 500 as closely as possible. Leave it to the professionals! Buy VOO or SPY and call it a day. But if you're willing to accept some tracking error, direct indexing is pretty cool. There are several nice advantages:

• Tax loss harvesting In other words, come tax time you can sell some of the losers to book a loss while buying other similar companies. With just a bit of python, you too can invest like Steve Ballmer!

• Limit positions Perhaps you believe that the S&P 500 is overweight on the MAG7 and you want to limit your exposure but don't want to go fully into an equal weighted index. (I don't personally do this but I think it is a valid consideration.)

• Your own personal ESG Personally, I don't want a guy who openly buys into nazi conspiracy theories and is also trying to shake down the board for money to be in control of a noticeable percentage of my portfolio. Maybe you have different priorities. ¯\_(ツ)_/¯ Direct indexing lets you make those priorities actionable, while still maintaining a broad index-based portfolio.

Full repo is on github.

1. Download all symbols and their weights in the S&P 500.

import pandas as pd
import requests
import cloudscraper
from io import StringIO
from bs4 import BeautifulSoup

pd.set_option('display.max_rows', 50)

url = "https://www.slickcharts.com/sp500"
scraper = cloudscraper.create_scraper()
soup = BeautifulSoup(scraper.get(url).text, 'html.parser')
sap_table = soup.find('table', {'class':"table"})
sap_list = pd.read_html(StringIO(str(sap_table)), flavor='html5lib')[0]
sap = pd.DataFrame(sap_list)

sap = sap[["Company", "Symbol", "Portfolio%"]]
sap

2. Remove any equities you don't want to hold

banned_tickers = ("TSLA", "FOX", "NWS")
sap = sap[~sap["Symbol"].isin(banned_tickers)].copy()
sap

3. Also optionally clip weights

sap["PortfolioFraction"] = sap["Portfolio%"].str.strip("%").astype('float') / 100
sap["PortfolioFraction"] = sap["PortfolioFraction"].clip(lower=0, upper=0.05)
sap

4. Reweight now that some of the tickers were removed or clipped

increase_prec = 1 - sap["PortfolioFraction"].sum() + 1
sap["PortfolioFraction"] = sap["PortfolioFraction"] * increase_prec
sap

5. Import a dataframe of your current equity holdings & cash. You're a bit on your own here but my brokerage allows me to download a CSV of all of my holding. I'm going to assume you have a simplified version of that.

#Write out some test data
with open('holdings.csv', 'w') as f:
    f.write("""symbol,shares
AMZN,10
NVDA,1
MSFT,1
YUM,100
ZTS,10
GME,100""")

holdings = pd.read_csv('holdings.csv')
holdings

cash_to_invest = 10000

6. Get last share price of all of holdings and compute our total assests to be invested

import yfinance as yf

def get_current_price(symbol):
    ticker = yf.Ticker(symbol.replace(".","-"))
    todays_data = ticker.history(period='2d')
    return todays_data['Close'].iloc[0]

holdings["last$"] = holdings["symbol"].apply(get_current_price)
holdings["total$"] = holdings["shares"] * holdings["last$"]

total_assets = holdings['total$'].sum() + cash_to_invest
total_assets
print(f"total_assets:", total_assets)
holdings

total_assets: 30298.690364837646

7. Also get lastest share price from all S&P 500 symbols

sap["last$"] = sap["Symbol"].apply(get_current_price)

sap

8. Compute shares we should own to match give our current investable assets

sap["$NeededToMatch"] = sap["PortfolioFraction"] * total_assets
sap["SharesToMatch"] = (sap["$NeededToMatch"] / sap["last$"]).astype('int') # assume no factional shares
sap

9. Join to our current holdings

movesWithHoldings = sap.merge(holdings[["symbol", "shares"]], left_on='Symbol', right_on='symbol', how='outer')
movesWithHoldings["shares"] = movesWithHoldings["shares"].fillna(0)
movesWithHoldings["SharesToMatch"] = movesWithHoldings["SharesToMatch"].fillna(0)
movesWithHoldings.loc[movesWithHoldings["Symbol"].isna(),"Symbol"] = movesWithHoldings[movesWithHoldings["Symbol"].isna()]["symbol"]
movesWithHoldings

11. Find the shares we need to match the S&P 500 prop

movesWithHoldings["SharesDifference"] = movesWithHoldings["SharesToMatch"] - movesWithHoldings["shares"]
movesWithHoldings

12. Find tickers that we need to sell

movesWithHoldings[movesWithHoldings["SharesDifference"] < 0][["Symbol", "SharesDifference"]]

13. Find tickers that we need to buy

movesWithHoldings[movesWithHoldings["SharesDifference"] > 0][["Symbol", "SharesDifference"]]