Anthropic, known for creating advanced chatbots that rival OpenAI’s ChatGPT, is facing accusations of aggressive data scraping from web publishers.
The company, founded by former OpenAI researchers with a mission to develop "responsible" AI systems, is alleged to have collected vast amounts of content from various websites to train its models, despite requests to stop.
Matt Barrie, CEO of Freelancer.com, claims Anthropic’s web crawler bombarded his site with 3.5 million visits in just four hours, ignoring standard web protocols and making the site slower for users.
Similar complaints were raised by other web publishers, including Kyle Wiens of iFixit.com, who reported a million hits from Anthropic bots in one day, setting off alarms and violating their terms of service.
Anthropic asserts that it respects web protocols like robots.txt and anti-circumvention technologies such as CAPTCHAs, aiming to minimize disruption.
However, the widespread and intensive scraping practices by AI companies, driven by the need to feed large language models, have increased costs and operational challenges for website operators.
As the AI industry tests the muddy waters of web data utilization, the ethical and operational impacts of such practices are becoming a contentious issue, highlighting the need for more responsible and cooperative approaches to data gathering.
Comments