i had a similar problem. yahoo doesn't offer it, but you can get one by looking through the document.write statements on nyse.com's list and finding the .js file where they just happen to store the list of companies starting with the given letter as a js array literal. you can also get nice tidy csv files from nasdaq.com here:
http://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=nasdaq&render=download (replace exchange=nasdaq with exchange=nyse for nyse symbols).
the last part of the URL is the date (in YYYYMMDD format) for which you want the
Earnings Calendar. You can loop through several days and scrape the Symbols
of all stocks that reported earnings on those days.
There is no guarantee that yahoo has data for all stocks that report earnings,
especially since some stocks no longer exist (bankruptcy, acquisition, etc.),
but this is probably a decent starting point.
If you are familiar with R, you can use the
qmao package to do this.
(See this post)
if you have trouble installing it.
ec <- getEarningsCalendar(from="2011-01-01", to="2012-07-01") #this may take a while
s <- unique(ec$Symbol)
length(s)
#[1] 12223
head(s, 20) #look at the first 20 Symbols
# [1] "CVGW" "ANGO" "CAMP" "LNDC" "MOS" "NEOG" "SONC"
# [8] "TISI" "SHLM" "FDO" "FC" "JPST.PK" "RECN" "RELL"
#[15] "RT" "UNF" "WOR" "WSCI" "ZEP" "AEHR"
This will not include any ETFs, futures, options, bonds, forex or mutual funds.
You can get a list of ETFs from yahoo here: http://finance.yahoo.com/etf/browser/mkt
That only shows the first 20. You need the URL of the "Show All" link at the
bottom of that page. You can scrape the page to find out how many
ETFs there are, then construct a URL.
L <- readLines("http://finance.yahoo.com/etf/browser/mkt")
# Sorry for the ugly regex
n <- gsub("^(\\w+)\\s?(.*)$", "\\1",
gsub("(.*)(Showing 1 - 20 of )(.*)", "\\3",
L[grep("Showing 1 - 20", L)]))
URL <- paste0("http://finance.yahoo.com/etf/browser/mkt?c=0&k=5&f=0&o=d&cs=1&ce=", n)
#http://finance.yahoo.com/etf/browser/mkt?c=0&k=5&f=0&o=d&cs=1&ce=1442
Now, you can extract the Tickers from the table on that page
library(XML)
tbl <- readHTMLTable(URL, stringsAsFactors=FALSE)
dat <- tbl[[tail(grep("Ticker", tbl), 1)]][-1, ]
colnames(dat) <- dat[1, ]
dat <- dat[-1, ]
etfs <- dat$Ticker # All ETF tickers from yahoo
length(etfs)
#[1] 1442
head(etfs)
#[1] "DGAZ" "TAGS" "GASX" "KOLD" "DWTI" "RTSA"
That's about all the help I can offer, but you could do something similar to
get some of the futures they offer by scraping these pages
(These are only U.S. futures)
There is a nice C# wrapper for the Yahoo.Finance API at http://code.google.com/p/yahoo-finance-managed/ that will get you there. Unfortunately there is no direct way to download the ticker list but the following creates the list by iterating through the alphabetical groups:
AlphabeticIDIndexDownload dl1 = new AlphabeticIDIndexDownload();
dl1.Settings.TopIndex = null;
Response<AlphabeticIDIndexResult> resp1 = dl1.Download();
writeStream.WriteLine("Id|Isin|Name|Exchange|Type|Industry");
foreach (var alphabeticalIndex in resp1.Result.Items)
{
AlphabeticalTopIndex topIndex = (AlphabeticalTopIndex) alphabeticalIndex;
dl1.Settings.TopIndex = topIndex;
Response<AlphabeticIDIndexResult> resp2 = dl1.Download();
foreach (var index in resp2.Result.Items)
{
IDSearchDownload dl2 = new IDSearchDownload();
Response<IDSearchResult> resp3 = dl2.Download(index);
int i = 0;
foreach (var item in resp3.Result.Items)
{
writeStream.WriteLine(item.ID + "|" + item.ISIN + "|" + item.Name + "|" + item.Exchange + "|" + item.Type + "|" + item.Industry);
}
}
}
It gave me a list of about 75,000 securities in about 4 mins.
I had same problem, but I think I have simple solution(code is from my RoR app):
Extract industry ids from yahoo.finance.sectors and add it to db:
select = "select * from yahoo.finance.sectors"
generate_query select
@data.each do |data|
data["industry"].each do |ind|
unless ind.kind_of?(Array)
unless ind["id"].nil?
id = ind["id"].to_i
if id > 0
Industry.where(id: id).first_or_create(name: ind["name"]).update_attribute(:name, ind["name"])
end
end
end
end
end
Extract all comanies with their symbols with industry ids:
ids = Industry.all.map{|ind| "'#{ind.id.to_s}'" }.join(",")
select = "select * from yahoo.finance.industry where id in"
generate_query select, ids
@data.each do |ts|
unless ts.kind_of?(Array) || ts["company"].nil?
if ts["company"].count == 2 && ts["company"].first[0] == "name"
t = ts["company"]
Ticket.find_or_create_by_symbol(symbol: t["symbol"], name: t["name"] ).update_attribute(:name, t["name"])
else
ts["company"].each do |t|
Ticket.find_or_create_by_symbol(symbol: t["symbol"], name: t["name"] ).update_attribute(:name, t["name"])
end
end
end
end
end
Connection hellper:
def generate_query(select, ids = nil)
if params[:form] || params[:action] == "sectors" || params[:controller] == "tickets"
if params[:action] == "sectors" || params[:controller] == "tickets"
if ids.nil?
query= select
else
query= "#{select} (#{ids})"
end
else
if params[:form][:ids]
@conditions = params_parse params[:form][:ids]
query = "#{select} (#{@conditions})"
end
end
yql_execut(query)
end
end
def yql_execut(query)
# TODO: OAuth ACCESS (http://developer.yahoo.com/yql/guide/authorization.html)
base_url = "http://query.yahooapis.com/v1/public/yql?&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&q="
dirty_data = JSON.parse(HTTParty.get(base_url + URI.encode(query)).body)
if dirty_data["query"]["results"] == nil
@data, @count, @table_head = nil
else
@data = dirty_data["query"]["results"].to_a[0][1].to_a
@count = dirty_data["query"]["count"]
if @count == 1
@table_head = @data.map{|h| h[0].capitalize}
else
@table_head = @data.to_a.first.to_a.map{|h| h[0].capitalize}
end
end
end
Sorry for mess, but this is first testing version for my project and I needed it very fast. There are some helpers variabels and other things for my app, sorry for it. But I have question: Have many symbols do you have? I have 5500.
It downloads a complete list of stock symbols using the Yahoo YQL API, including the stock name, stock symbol, and industry ID. What it doesn't seem to have is any sort of stock symbol modifiers. E.g. for Rogers Communications Inc, it only downloads RCI, not RCI-A.TO, RCI-B.TO, etc. I haven't found a source for that information yet - if anyone knows of a way to automate downloading that, I'd like to hear it. Also, it'd be nice to find a way to download some sort of relation between the stock symbol and the exchange it's traded on, since some are traded on multiple exchanges, or maybe I only want to look at stuff on the TSX or something.
I have been researching this for a few days, following endless leads that got close, but not quite, to what I was after.
My need is for a simple list of 'symbol, sector, industry'. I'm working in Java and don't want to use any platform native code.
It seems that most other data, like quotes, etc., is readily available.
Finally, followed a suggestion to look at 'finviz.com'. Looks like just the ticket. Try using the following:
http://finviz.com/export.ashx?v=111&t=aapl,cat&o=ticker
This comes back as lines, csv style, with a header row, ordered by ticker symbol. You can keep adding tickers. In code, you can read the stream. Or you can let the browser ask you whether to open or save the file.
Replace 'export' with 'screener' and the data will show up in the browser.
There are many more options you can use, one for every screener element on the site.
So far, this is the most powerful and convenient programmatic way to get the few pieces of data I couldn't otherwise seem to easily get. And, it looks like this site could well be a single source for most of what you might need other than real- or near-real-time quotes.