-
Multiple Vulnerabilities in Fortinet Products Could Allow for Arbitrary Code Execution
Multiple vulnerabilities have been discovered in Fortinet Products, the most severe of which could allow for arbitrary code execution. The products affected by vulnerabilities in this round of monthly Fortinet patches is: FortiADC is an application delivery controller (ADC) from Fortinet that enhances application availability, performance, and security. It offers features like load balancing,…
-
How TimescaleDB helped us scale analytics and reporting
At Cloudflare, PostgreSQL and ClickHouse are our standard databases for transactional and analytical workloads. If you’re part of a team building products with configuration in our Dashboard, chances are you’re using PostgreSQL. It’s fast, versatile, reliable, and backed by over 30 years of development and real-world use. It has been a foundational part of our…
-
Critical Patches Issued for Microsoft Products, May 13, 2025
Multiple vulnerabilities have been discovered in Microsoft products, the most severe of which could allow for remote code execution in the context of the logged on user. Depending on the privileges associated with the user, an attacker could then install programs; view, change, or delete data; or create new accounts with full user rights. Users…
-
Introducing simple and secure egress policies by hostname in Cloudflare’s SASE platform
Cloudflare’s SASE platform is on a mission to strengthen our platform-wide support for hostname- and domain-based policies. This mission is being driven by enthusiastic demands from our customers, and boosted along the way by several interesting engineering challenges. Today, we’re taking a deep dive into the first milestone of this mission, which we recently released…
-
Content Independence Day: no AI crawl without compensation!
Almost 30 years ago, two graduate students at Stanford University — Larry Page and Sergey Brin — began working on a research project they called Backrub. That, of course, was the project that resulted in Google. But also something more: it created the business model for the web. The deal that Google made with content…
-
The crawl before the fall… of referrals: understanding AI’s impact on content providers
Content publishers welcomed crawlers and bots from search engines because they helped drive traffic to their sites. The crawlers would see what was published on the site and surface that material to users searching for it. Site owners could monetize their material because those users still needed to click through to the page to access…
-
Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
Cloudflare is giving all website owners two new tools to easily control whether AI bots are allowed to access their content for model training. First, customers can let Cloudflare create and manage a robots.txt file, creating the appropriate entries to let crawlers know not to access their site for AI training. Second, all customers can…
-
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
A changing landscape of consumption Many publishers, content creators and website owners currently feel like they have a binary choice — either leave the front door wide open for AI to consume everything they create, or create their own walled garden. But what if there was another way? At Cloudflare, we started from a simple…
-
From Googlebot to GPTBot: who’s crawling your site in 2025
Web crawlers are not new. The World Wide Web Wanderer debuted in 1993, though the first web search engines to truly use crawlers and indexers were JumpStation and WebCrawler. Crawlers are part of one of the backbones of the Internet’s success: search. Their main purpose has been to index the content of websites across the…
-
Message Signatures are now part of our Verified Bots Program, simplifying bot authentication
As a site owner, how do you know which bots to allow on your site, and which you’d like to block? Existing identification methods rely on a combination of IP address range (which may be shared by other services, or change over time) and user-agent header (easily spoofable). These have limitations and deficiencies. In our…