State of the industry
Most stores have limited searches, no access to (or do not provide access to) all the data
Some stores are crawlable, i.e. all data is accessible
Some stores expose dedicated URLs, files, or XML types interfaces dedicated for bots
Some stores provide direct access to product catalogs and product pages
In general difficult to block bots; easy to simulate browser requests; some ethics/policies are emerging