• bitcoinBitcoin(BTC)$58,358.00-5.77%
  • ethereumEthereum(ETH)$1,598.87-8.91%
  • tetherTether(USDT)$1.00-0.06%
  • binancecoinBNB(BNB)$533.63-7.92%
  • usd-coinUSDC(USDC)$1.00-0.02%
  • rippleXRP(XRP)$0.89-7.52%
  • solanaSolana(SOL)$67.44-7.41%
  • tronTRON(TRX)$0.300847-2.93%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.02-2.11%
  • HyperliquidHyperliquid(HYPE)$60.30-10.72%
  • zcashZcash(ZEC)$736.37-17.30%
  • moneroMonero(XMR)$448.79-9.02%
  • leo-tokenLEO Token(LEO)$9.16-1.21%
  • dogecoinDogecoin(DOGE)$0.052796-9.06%
  • whitebitWhiteBIT Coin(WBT)$55.87-7.11%
  • chainlinkChainlink(LINK)$8.32-6.99%
  • USDSUSDS(USDS)$1.000.06%
  • cardanoCardano(ADA)$0.139326-7.86%
  • Ethena USDeEthena USDe(USDE)$1.00-0.07%
  • stellarStellar(XLM)$0.125401-5.33%
  • daiDai(DAI)$1.000.03%
  • bitcoin-cashBitcoin Cash(BCH)$189.04-8.38%
  • nearNEAR Protocol(NEAR)$2.78-14.18%
  • litecoinLitecoin(LTC)$39.74-5.59%
  • CantonCanton(CC)$0.073563-8.73%
  • USD1USD1(USD1)$1.00-0.07%
  • avalanche-2Avalanche(AVAX)$6.45-7.07%
  • Ripple USDRipple USD(RLUSD)$1.000.02%
  • tether-goldTether Gold(XAUT)$3,516.27-3.66%
  • the-open-networkGram (prev. Toncoin)(GRAM)$0.98-4.23%
  • uniswapUniswap(UNI)$4.36-12.29%
  • Global DollarGlobal Dollar(USDG)$1.000.01%
  • suiSui(SUI)$0.64-10.04%
  • RainRain(RAIN)$0.003644-19.71%
  • paypal-usdPayPal USD(PYUSD)$1.000.05%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.11%
  • hedera-hashgraphHedera(HBAR)$0.050527-6.33%
  • shiba-inuShiba Inu(SHIB)$0.000004-7.49%
  • BittensorBittensor(TAO)$175.39-8.79%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.08%
  • crypto-com-chainCronos(CRO)$0.039044-4.97%
  • Circle USYCCircle USYC(USYC)$1.140.05%
  • quant-networkQuant(QNT)$135.27-9.88%
  • MemeCoreMemeCore(M)$0.852.48%
  • Falcon USDFalcon USD(USDF)$1.00-0.08%
  • United StablesUnited Stables(U)$1.00-0.04%
  • okbOKB(OKB)$77.51-9.82%
  • pax-goldPAX Gold(PAXG)$3,516.04-3.64%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • aaveAave(AAVE)$95.13-8.73%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Google Joins OpenAI And Anthropic In Promising New Scraping Controls

October 31, 2026
in AI & Technology
Reading Time: 2 mins read
A A
Google Joins OpenAI And Anthropic In Promising New Scraping Controls
ShareShareShareShareShare

The new robots.txt directive will prevent the companies from scraping content for generative AI training.



tadamichi/Shutterstock

Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley. 

YOU MAY ALSO LIKE

This GTA 6 Fast Travel Trick Comes At A Price

RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only

As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley’s proposal will change that by getting multiple AI companies to recognize and honor a single common signal. Publishers have to set up a new user agent in the robots.txt file that’s specifically for AI training, which will look like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That will block crawlers specifically meant to gather data for AI training. The “Allow” section gives website operators the option to allow scraping for AI training for specific parts of the page, such as if there’s a press kit on it. But this new user-agent section doesn’t prevent crawling altogether or prevent content from showing up on Google search. For instance, Google will specifically interpret it as a stop sign specifically for the “Google-Extended” user agent, which crawls websites for materials used to train Gemini models. 

Another thing to note is that this directive cannot undo any past scraping the companies have already done. If they’d crawled a page previously, they already have its content. This will just prevent them from scraping websites or specific pages in the future. Anthropic has already started supporting the new directive, while Microsoft said it will start supporting it on February 1, 2027. Google said it will support it in early 2027, as well, while OpenAI pledged to implement the new signal in the coming months. 

Credit: Source link

ShareTweetSendSharePin

Related Posts

This GTA 6 Fast Travel Trick Comes At A Price
AI & Technology

This GTA 6 Fast Travel Trick Comes At A Price

November 20, 2026
RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only
AI & Technology

RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only

November 20, 2026
Andrew Nassar, CTO of IC Realtime – Interview Series – Unite.AI
AI & Technology

Andrew Nassar, CTO of IC Realtime – Interview Series – Unite.AI

November 20, 2026
You Can Reveal The Full GTA 6 Map In Minutes Using This Simple Plane Trick
AI & Technology

You Can Reveal The Full GTA 6 Map In Minutes Using This Simple Plane Trick

November 20, 2026
Next Post
Meet Factory Droid 1.0: A Model-Independent Coding Agent Harness Built on Script Mode

Meet Factory Droid 1.0: A Model-Independent Coding Agent Harness Built on Script Mode

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
My Top Dividend Growth Picks In Each GICS Sector (November 2026)

My Top Dividend Growth Picks In Each GICS Sector (November 2026)

November 20, 2026
Iranian art installation puts Trump in a cage

Iranian art installation puts Trump in a cage

November 19, 2026
Holiday Spending Cap – Set It at What You Can Pay Off in January

Holiday Spending Cap – Set It at What You Can Pay Off in January

November 15, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!