The National Association of Realtors is hoping to smooth over a flap that erupted when it backed a local Realtor board’s decision to force one of its brokers to prevent search engines like Google from indexing other brokers’ for-sale listings.
Many real estate brokers display not only their own listings on their Web sites, but those of other members of their multiple listings service (MLS) through Internet Data Exchange (IDX) reciprocity agreements.
NAR has an IDX policy that requires brokers participating in such reciprocity agreements to "protect IDX information from misappropriation by employing reasonable efforts to monitor and to prevent ‘scraping’ or other unauthorized accessing, reproduction, or use of the MLS database."
Responding to a complaint from one of its members, the Metropolitan Indianapolis Board of Realtors (MIBOR) last month sent real estate agent Paula Henry and her broker cease-and-desist letters demanding that they take steps to prevent other brokers’ listings from being indexed by Google and other search engines.
MIBOR says it acted after consulting with NAR on its IDX policy, and being advised that it could require members to use the "proper technology" to prevent IDX listings from being indexed by search engines. One way to prevent indexing is to employ robot.txt files, which direct automated search-engine "spiders" to ignore specified files or directories.
"Nobody is calling search engines scrapers, but the (policy) requires the IDX broker to prevent unauthorized use," said Tom Renkert, MIBOR’s information services director.
When search-engine spiders crawl sites, they collect information and add it to internal databases that make search results faster and more relevant. The public can sometimes access those databases, through features such as Google "cache."
"The theory is that in the indexing process, the search engine reads through the Web page and pulls out information it writes to its database," including details of listings from the MLS, Renkert said.
Search engines aren’t the only sites that use spiders to crawl Web sites for information — third-party listing sites that rely on "scraping" can also access IDX data that’s not protected from indexing.
Renkert said MIBOR received a complaint from a member broker that "another broker is advertising my listings without my permission, and that the source was the IDX database."
Although Henry disputes that she violated any MLS rules, she said she and her broker decided to comply with the request, because "it’s hard to fight an 800-pound gorilla."
Henry said she has about 28,000 property listings on her IDX Web site, which are "embedded" rather than "framed," allowing Google to read the pages.
After removing property details from "title tags" — coding search engines use to categorize content but which is invisible to Web surfers — Henry said MIBOR then demanded that property descriptions be stripped of all MLS data, including address, MLS number, neighborhood, subdivision, price and property type.
"Start your newspaper ad with your client’s home featured in black and white and gather up your open house signs, we’re heading backwards," Henry complained in a post on the blog Agent Genius. "Let’s hide the information the client wants, put our MLS listings in a book and wait a week for them to be published. Let’s force the hand of the client to only have access to the agent who listed the home."
Henry’s post on Agent Genius sparked a wave of complaints from brokers and agents who said MIBOR and NAR came down on the wrong side of the issue. …CONTINUED
"I agree that there needs to be some rules," said Phoenix-based broker-owner Jay Thompson, in one of the more than 350 comments responding to Henry’s post. "This isn’t, after all, the Wild West where anarchy rules."
But equating scrapers with search engines is "ridiculous," Thompson said. Instead of saying, " ‘Huh, the intent of the policy is to stop malicious use of the data, how do we do that?’ we now have, ‘Nothing can be indexed!’ " Thompson complained.
While some brokers and agents may object to search engines indexing IDX listings, others see it as a vital part of boosting their search-engine rankings.
A recent white paper from the WAV Group consulting firm, "Geo-Domain Targeting trending to assist Real Estate Marketing," concluded that exposing listings to search-engine spiders can be an important part of a strategy to help brokers and agents compete with big Web sites like Trulia and Zillow (see story).
WAV Group partner Victor Lund noted that not all MLSs have interpreted NAR’s IDX policy in regards to indexing as strictly as MIBOR, but said agents and brokers would be at a disadvantage if they did.
"I believe that it would be a blow to agents and brokers who are trying to maintain a voice to the consumer if the enforcement of this rule moves forward," Lund said. "Simply stated, it would provide an advantage to third-party Web sites like Zillow, Trulia and others to gain a foothold in search and block the listing owners from competing."
In her Agent Genius post, Henry complained that while MIBOR won’t allow her to let search engines index IDX listings, those same listings are sent to Realtor.com, where they are indexed.
Renkert pointed out that while brokers can opt out of sending listings to Realtor.com, they have less control over what happens to listings shared through IDX reciprocity agreements. Although brokers can choose not to participate, he said MIBOR enjoys 100 percent IDX participation among its 6,500 members.
"The question is, does the IDX policy grant the IDX broker the ability to advertise those listings in search engines that chose to index those Web pages," Renkert said. "Or are they required to use (steps like the) robot.txt file … to prevent robots from scanning through the information?"
Cliff Niersbach, NAR’s vice president of board policy and programs, issued a statement acknowledging that NAR’s interpretation of the IDX policy — by staffers at the Center for Realtor Technology — "hasn’t sat well with some IDX site operators and some IDX vendors."
But Niersbach said NAR staff interpretation of the policy — last revised in May 2005 — "isn’t the last word." If Realtors think the IDX policy should be changed to permit indexing while continuing to prohibit scraping, they can petition NAR’s Multiple Listing Issues and Policies Committee, he said.
The IDX policy is already being looked at, he said, "with an eye toward incorporating certain aspects of the recently revised VOW policy, and distinguishing between scraping and indexing could be part of the effort."
As part of a settlement of a Justice Department antitrust lawsuit, NAR last year adopted a policy allowing member brokers that participate in MLSs to operate "Virtual Office Web sites," or VOWs, that provide a broader range of property information than typically found on public-facing MLS sites (see story).
The MLS and Policies Committee meets tomorrow at NAR’s midyear conference in Washington, D.C. In order to make sure the committee hears the voice of affected agents and brokers, NAR invited Henry and Thompson to come to Washington, D.C., and speak to it. Both accepted, and NAR is paying their travel expenses.
"This will not be the dominant topic discussed, but Jay and Paula will be able to bring this issue to the committee’s attention," said NAR’s social media manager, Todd Carpenter, in one of several responses to Henry’s Agent Genius post.
Renkert said MIBOR’s board hasn’t taken a position on whether the MLS and Policies Committee should amend the IDX policy to allow indexing. But he said it might be difficult to draft a policy that permits search engines to index IDX listings but bars "scraping" by third-party sites.
"One concern is how do you define the difference between a scraper and indexer?" Renkert said.
***
What’s your opinion? Leave your comments below or send a letter to the editor.
