I needed to complete some analysis on registered domains. Appears like I'm able to hit whois.internic.internet to obtain details about each domain, it appears like you will find rate limits that prevent me from doing large amounts of queries.

It is possible to method to periodically (say daily) grab the whole whois database? I truly only worry about whether a website is registered or otherwise, and so i have no need for the entire whois information.

For .com and .internet, Verisign offers the TLD Zone Access Program where one can make an application for download use of the zone files. Other registrars usually provide similar services.

Here's a listing to many other TLDs (though I've not attempted to try to get these):

Whois API provides the entire whois database download in main GTLDs(.com,.internet,.org,.us,.biz.,.mobi,etc)
In addition, it offers aged historic whois database both in parsed and raw format for download as CSV files; I used an incomplete database download for an organization project(Search engine optimization related) and also the data quality was very good.

http://world wide web.whoisxmlapi.com/whois-database-download.php

These men will sell you monthly use of the main zone files (plus lists of changes):

http://world wide web.premiumdrops.com/zones.html

In 2006 Dennis Forbes did some analysis you can observe much more about it here. Younger crowd mentions within the sidebar where he got the information. Appears he was searching at Verisign zone files as opposed to the whois database. I do not the particulars but this can be another direction to appear in.