×
The executive that helped build Meta’s ad machine is trying to expose it

The executive that helped build Meta’s ad machine is trying to expose it

Brian Boland spent more than a decade figuring out how to build a system that would make Meta money. On Thursday, he told a California jury it incentivized drawing more and more users, including teens, onto Facebook and Instagram — despite the risks.

Boland’s testimony came a day after Meta CEO Mark Zuckerberg took the stand in a case over whether Meta and YouTube are liable for allegedly harming a young woman’s mental health. Zuckerberg framed Meta’s mission as balancing safety with free expression, not revenue. Boland’s role was to counter this by explaining how Meta makes money, and how that shaped its platforms’ design. Boland testified that Zuckerberg fostered a culture that prioritized growth and profit over users’ wellbeing from the top down. He said he’s been described as a whistleblower — a term Meta has broadly sought to limit for fear it would prejudice the jury, but which the judge has generally allowed. Over his 11 years at Meta, Boland said he went from having “deep blind faith” in the company to coming to the “firm belief that competition and power and growth were the things that Mark Zuckerberg cared about most.”

Boland last served as Meta’s VP of partnerships before leaving in 2020, working to bring content to the platform that it could monetize, and previously worked in a variety of advertising roles beginning in 2009. He testified that Facebook’s infamous early slogan of “move fast and break things” represented “a cultural ethos at the company.” He said the idea behind the motto was generally, “don’t really think about what could go wrong with a product, but just get it out there and learn and see.” At the height of its prominence internally, employees would sit down at their desks to see a piece of paper that said, “what will you break today?” Boland testified.

“The priorities were on winning growth and engagement”

Zuckerberg consistently made his priorities for the company abundantly clear, according to Boland. He’d announce them in all hands meetings and leave no shadow of a doubt what the company should be focused on, whether it was building its products to be mobile-first, or getting ahead of the competition. When Zuckerberg realized that then-Facebook had to get into shape to compete with a rumored Google social network competitor (which he didn’t name, but seemed to refer to Google+), Boland recalled a digital countdown clock in the office that symbolized how much time they had left to achieve their goals during what the company called a “lockdown.” During his time at the company, Boland testified, there was never a lockdown around user safety, and Zuckerberg allegedly instilled in engineers that “the priorities were on winning growth and engagement.”

Meta has repeatedly denied that it tries to maximize users’ engagement on its platforms over safeguarding their wellbeing. In the past weeks, both Zuckerberg and Instagram CEO Adam Mosseri testified that building platforms that users enjoy and feel good on is in their long-term interest, and that’s what drives their decisions.

Boland disputes this. “My experience was that when there were opportunities to really try to understand what the products might be doing harmfully in the world, that those were not the priority,” he testified. “Those were more of a problem than an opportunity to fix.”

When safety issues came up through press reports or regulatory questions, Boland said, “the primary response was to figure out how to manage through the press cycle, to what the media was saying, as opposed to saying, ‘let’s take a step back and really deeply understand.” Though Boland said he told his advertising-focused team that they should be the ones to discover “broken parts,” rather than those outside the company, he said that philosophy didn’t extend to the rest of the company.

On the stand the day before, Zuckerberg pointed to documents around 2019 showing disagreement among his employees with his decisions, saying they demonstrated a culture that encourages a diversity of opinion. Boland, however, testified that while that might have been the case earlier in his tenure, it later became “a very closed down culture.”

“There’s not a moral algorithm, that’s not a thing … Doesn’t eat, doesn’t sleep, doesn’t care”

Since the jury can only consider decisions and products that Meta itself made, rather than content it hosted from users, lead plaintiff attorney Mark Lanier also had Boland describe how Meta’s algorithm works, and the decisions that went into making and testing it. Algorithms have an “immense amount of power,” Boland said, and are “absolutely relentless” in pursuing their programmed goals — in many cases at Meta, that was allegedly engagement. “There’s not a moral algorithm, that’s not a thing,” Boland said. “Doesn’t eat, doesn’t sleep, doesn’t care.”

During his testimony on Wednesday, Zuckerberg commented that Boland “developed some strong political opinions” toward the end of his time at the company. (Neither Zuckerberg nor Boland offered specifics, but in a 2025 blog post, Boland indicated he was deleting his Facebook account in part over disagreements with how Meta handled events like January 6th, writing that he believed “Facebook had contributed to spreading ‘Stop the Steal’ propaganda and enabling this attempted coup.”) Lanier spent time establishing that Boland was respected by peers, showing a CNBC article about his departure that quoted a glowing statement from his then-boss, and a reference to an unnamed source who reportedly described Boland as someone with a strong moral character.

On cross examination, Meta attorney Phyllis Jones clarified that Boland didn’t work on the teams tasked with understanding youth safety at the company. Boland agreed that advertising business models are not inherently bad, and neither are algorithms. He also admitted that many of his concerns involved the content users were posting, which is not relevant to the current case.

During his direct examination, Lanier asked if Boland had ever expressed his concerns to Zuckerberg directly. Boland said he’d told the CEO he’d seen concerning data showing “harmful outcomes” of the company’s algorithms and suggested that they investigate further. He recalled Zuckerberg responding something to the effect of, “I hope there’s still things you’re proud of.” Soon after, he said, he quit.

Boland said he left upwards of $10 million worth of unvested Meta stock on the table when he departed, though he admitted he made more than that over the years. He said he still finds it “nerve-wracking” every time he speaks out about the company. “This is an incredibly powerful company,” he said.

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.


Source link
#executive #helped #build #Metas #machine #expose

For some social media users, Facebook may feel like an app left behind in an online era of the past. However, for others, Facebook has reinvented itself as the modern-day Craigslist with Facebook Marketplace.

Meta seems to have taken notice and is now releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.

The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 

Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.

Meta launches Seller, a standalone app for Facebook Marketplace sellers
                                                            For some social media users, Facebook may feel like an app left behind in an online era of the past. However, for others, Facebook has reinvented itself as the modern-day Craigslist with Facebook Marketplace.Meta seems to have taken notice and is now releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 
Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.
        
            Mashable Light Speed
        
        
    

    
                    


            
            
            
            Credit: Facebook
        
    
Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.
As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

                    
                                            
                            
                        
                                    #Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers

Credit: Facebook

Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. 

Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.

In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.

As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.

Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

#Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers">Meta launches Seller, a standalone app for Facebook Marketplace sellers
                                                            For some social media users, Facebook may feel like an app left behind in an online era of the past. However, for others, Facebook has reinvented itself as the modern-day Craigslist with Facebook Marketplace.Meta seems to have taken notice and is now releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 
Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.
        
            Mashable Light Speed
        
        
    

    
                    


            
            
            
            Credit: Facebook
        
    
Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.
As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

                    
                                            
                            
                        
                                    #Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers

releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.

The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 

Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.

Meta launches Seller, a standalone app for Facebook Marketplace sellers
                                                            For some social media users, Facebook may feel like an app left behind in an online era of the past. However, for others, Facebook has reinvented itself as the modern-day Craigslist with Facebook Marketplace.Meta seems to have taken notice and is now releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 
Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.
        
            Mashable Light Speed
        
        
    

    
                    


            
            
            
            Credit: Facebook
        
    
Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.
As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

                    
                                            
                            
                        
                                    #Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers

Credit: Facebook

Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. 

Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.

In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.

As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.

Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

#Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers">Meta launches Seller, a standalone app for Facebook Marketplace sellers

For some social media users, Facebook may feel like an app left behind in an online era of the past. However, for others, Facebook has reinvented itself as the modern-day Craigslist with Facebook Marketplace.

Meta seems to have taken notice and is now releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.

The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 

Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.

Meta launches Seller, a standalone app for Facebook Marketplace sellers
                                                            For some social media users, Facebook may feel like an app left behind in an online era of the past. However, for others, Facebook has reinvented itself as the modern-day Craigslist with Facebook Marketplace.Meta seems to have taken notice and is now releasing a standalone app for the Facebook Marketplace power users who keep the platform going: It’s sellers.The new app, aptly named Seller, launched on Friday and provides Facebook Marketplace sellers with tools to list their wares and manage their current listings. 
Like Facebook Marketplace itself, the Seller app is free for users. Meta monetizes Facebook Marketplace with ads and does not take any percentage of its users’ sales. Users will be able to download the app and login with their Facebook account starting today.
        
            Mashable Light Speed
        
        
    

    
                    


            
            
            
            Credit: Facebook
        
    
Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.
As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

                    
                                            
                            
                        
                                    #Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers

Credit: Facebook

Seller provides a tool to post new listings, a dashboard to manage their existing ones, an in-app inbox to reply to prospective buyers, and analytics for all of their old listings. 

Seller also introduces a few new tools for sellers as well. According to Facebook, Seller users will be able to scan their products with their smartphone, and AI will fill-out the listing details automatically.

In an interview with the New York Times, Facebook head Tom Allison said that the app would be the Marketplace seller equivalent of a video editing tool for content creators. Allison also explained how Meta is experimenting with more standalone products for Facebook’s power users, which include Marketplace Sellers and Facebook Groups administrators.

As social media users move to competing platforms like TikTok and Meta’s own Instagram, Facebook has found new life within specific, once-niche features on the platform. In May, Facebook launched a standalone app called Forum for its Facebook Groups power users. Now, the company is doing the same for Facebook Marketplace sellers.

Meta says that one-third of young adults who use Facebook also use Marketplace. And, every month, more than 420 million items are listed on the Marketplace platform. 

#Meta #launches #Seller #standalone #app #Facebook #Marketplace #sellers
For months, AI giants have devised special vetted programs and strict guardrails to limit the use of their models by malicious hackers. But these limits are now hindering the work of legitimate network defenders, as well as that of offensive cybersecurity researchers. 

In June, the U.S. government slapped export control restrictions on Anthropic’s much-hyped AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ guardrails designed to prevent users from using them to build and execute malicious cyberattacks.

Regardless of whether the incident was really motivated by fears of a jailbreak, the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place. (The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government’s review process.)

That kind of gatekeeping isn’t unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and — if approved — access models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber program and Anthropic’s Cyber Verification Program

These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.

During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said that, “it’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Dowd has spent decades finding and selling “zero-days” — previously unknown software flaws and the exploits that take advantage of them — to Western governments, rather than reporting them to the software makers so they get patched. Governments pay a premium for vulnerabilities precisely because they stay open, which is useful for intelligence operations.

Dowd admitted his work may make him biased, but he isn’t alone. Several people who work in offensive cybersecurity — they proactively probe systems for weaknesses — described to TechCrunch how they use AI tools and deal with their guardrails. 

Chris Anley, the chief scientist at security consulting giant NCC Group, said that asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing. But if a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders, he said.

“This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” said Anley. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked.”

It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

When he and his colleagues run into such a roadblock, they sometimes fall back on open source AI models that come with no guardrails at all.

Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails. 

Stagno said he and his colleagues do use frontier models — but only for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits, he said, because feeding that work into a cloud-based model risks leaking sensitive vulnerability data or having it absorbed into future training runs. For that step, he said, they use open source models run locally, as they do not rely on sharing data outside of the model. 

Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails are not impeding his work. That’s because he doesn’t use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he’s analyzing, and to build supporting tools. For that, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities. 

“I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” said Cali. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he isn’t authorized to talk to the press, said his employer isn’t part of Anthropic’s CVP program and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict.

“If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said. 

Chris Thompson — chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event — said that in his experience using the frontier AI models, the guardrails can be inconsistent and work differently every day. That’s true even inside the looser boundaries of Anthropic’s and OpenAI’s vetted programs. 

“I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” said Thompson. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.” 

Consequently, researchers rely on or get pushed toward Chinese open source models like GLM — freely downloadable models that can be run locally with no vetting or usage restrictions — said Thompson.

“You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

Rather than tightening restrictions further, Thompson called for the AI frontier labs to open up their programs, provide responsible access, and hold those who abuse their tools accountable. Otherwise, he argued, defenders will lose the AI race.

“There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” said Thompson. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.”

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

#guardrails #impeding #work #offensive #cybersecurity #researchers #TechCrunchcybersecurity,Zero-days">How AI guardrails are impeding the work of offensive cybersecurity researchers | TechCrunch
For months, AI giants have devised special vetted programs and strict guardrails to limit the use of their models by malicious hackers. But these limits are now hindering the work of legitimate network defenders, as well as that of offensive cybersecurity researchers. 

In June, the U.S. government slapped export control restrictions on Anthropic’s much-hyped AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ guardrails designed to prevent users from using them to build and execute malicious cyberattacks.







Regardless of whether the incident was really motivated by fears of a jailbreak, the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place. (The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government’s review process.)

That kind of gatekeeping isn’t unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and — if approved — access models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber program and Anthropic’s Cyber Verification Program. 

These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.

During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said that, “it’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Dowd has spent decades finding and selling “zero-days” — previously unknown software flaws and the exploits that take advantage of them — to Western governments, rather than reporting them to the software makers so they get patched. Governments pay a premium for vulnerabilities precisely because they stay open, which is useful for intelligence operations.


Dowd admitted his work may make him biased, but he isn’t alone. Several people who work in offensive cybersecurity — they proactively probe systems for weaknesses  — described to TechCrunch how they use AI tools and deal with their guardrails. 

Chris Anley, the chief scientist at security consulting giant NCC Group, said that asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing. But if a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders, he said. 

“This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” said Anley. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked.” 







It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

When he and his colleagues run into such a roadblock, they sometimes fall back on open source AI models that come with no guardrails at all.

Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails. 

Stagno said he and his colleagues do use frontier models — but only for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits, he said, because feeding that work into a cloud-based model risks leaking sensitive vulnerability data or having it absorbed into future training runs. For that step, he said, they use open source models run locally, as they do not rely on sharing data outside of the model. 

Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails are not impeding his work. That’s because he doesn’t use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he’s analyzing, and to build supporting tools. For that, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities. 

“I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” said Cali. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he isn’t authorized to talk to the press, said his employer isn’t part of Anthropic’s CVP program and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict.

“If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said. 







Chris Thompson — chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event — said that in his experience using the frontier AI models, the guardrails can be inconsistent and work differently every day. That’s true even inside the looser boundaries of Anthropic’s and OpenAI’s vetted programs. 

“I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” said Thompson. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.” 

Consequently, researchers rely on or get pushed toward Chinese open source models like GLM — freely downloadable models that can be run locally with no vetting or usage restrictions — said Thompson.

“You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

Rather than tightening restrictions further, Thompson called for the AI frontier labs to open up their programs, provide responsible access, and hold those who abuse their tools accountable. Otherwise, he argued, defenders will lose the AI race.

“There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” said Thompson. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.”


When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.#guardrails #impeding #work #offensive #cybersecurity #researchers #TechCrunchcybersecurity,Zero-days

slapped export control restrictions on Anthropic’s much-hyped AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ guardrails designed to prevent users from using them to build and execute malicious cyberattacks.

Regardless of whether the incident was really motivated by fears of a jailbreak, the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place. (The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government’s review process.)

That kind of gatekeeping isn’t unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and — if approved — access models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber program and Anthropic’s Cyber Verification Program

These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.

During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said that, “it’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Dowd has spent decades finding and selling “zero-days” — previously unknown software flaws and the exploits that take advantage of them — to Western governments, rather than reporting them to the software makers so they get patched. Governments pay a premium for vulnerabilities precisely because they stay open, which is useful for intelligence operations.

Dowd admitted his work may make him biased, but he isn’t alone. Several people who work in offensive cybersecurity — they proactively probe systems for weaknesses — described to TechCrunch how they use AI tools and deal with their guardrails. 

Chris Anley, the chief scientist at security consulting giant NCC Group, said that asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing. But if a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders, he said.

“This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” said Anley. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked.”

It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

When he and his colleagues run into such a roadblock, they sometimes fall back on open source AI models that come with no guardrails at all.

Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails. 

Stagno said he and his colleagues do use frontier models — but only for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits, he said, because feeding that work into a cloud-based model risks leaking sensitive vulnerability data or having it absorbed into future training runs. For that step, he said, they use open source models run locally, as they do not rely on sharing data outside of the model. 

Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails are not impeding his work. That’s because he doesn’t use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he’s analyzing, and to build supporting tools. For that, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities. 

“I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” said Cali. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he isn’t authorized to talk to the press, said his employer isn’t part of Anthropic’s CVP program and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict.

“If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said. 

Chris Thompson — chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event — said that in his experience using the frontier AI models, the guardrails can be inconsistent and work differently every day. That’s true even inside the looser boundaries of Anthropic’s and OpenAI’s vetted programs. 

“I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” said Thompson. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.” 

Consequently, researchers rely on or get pushed toward Chinese open source models like GLM — freely downloadable models that can be run locally with no vetting or usage restrictions — said Thompson.

“You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

Rather than tightening restrictions further, Thompson called for the AI frontier labs to open up their programs, provide responsible access, and hold those who abuse their tools accountable. Otherwise, he argued, defenders will lose the AI race.

“There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” said Thompson. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.”

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

#guardrails #impeding #work #offensive #cybersecurity #researchers #TechCrunchcybersecurity,Zero-days">How AI guardrails are impeding the work of offensive cybersecurity researchers | TechCrunch

For months, AI giants have devised special vetted programs and strict guardrails to limit the use of their models by malicious hackers. But these limits are now hindering the work of legitimate network defenders, as well as that of offensive cybersecurity researchers. 

In June, the U.S. government slapped export control restrictions on Anthropic’s much-hyped AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ guardrails designed to prevent users from using them to build and execute malicious cyberattacks.

Regardless of whether the incident was really motivated by fears of a jailbreak, the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place. (The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government’s review process.)

That kind of gatekeeping isn’t unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and — if approved — access models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber program and Anthropic’s Cyber Verification Program

These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.

During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said that, “it’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Dowd has spent decades finding and selling “zero-days” — previously unknown software flaws and the exploits that take advantage of them — to Western governments, rather than reporting them to the software makers so they get patched. Governments pay a premium for vulnerabilities precisely because they stay open, which is useful for intelligence operations.

Dowd admitted his work may make him biased, but he isn’t alone. Several people who work in offensive cybersecurity — they proactively probe systems for weaknesses — described to TechCrunch how they use AI tools and deal with their guardrails. 

Chris Anley, the chief scientist at security consulting giant NCC Group, said that asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing. But if a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders, he said.

“This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” said Anley. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked.”

It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

When he and his colleagues run into such a roadblock, they sometimes fall back on open source AI models that come with no guardrails at all.

Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails. 

Stagno said he and his colleagues do use frontier models — but only for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits, he said, because feeding that work into a cloud-based model risks leaking sensitive vulnerability data or having it absorbed into future training runs. For that step, he said, they use open source models run locally, as they do not rely on sharing data outside of the model. 

Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails are not impeding his work. That’s because he doesn’t use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he’s analyzing, and to build supporting tools. For that, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities. 

“I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” said Cali. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he isn’t authorized to talk to the press, said his employer isn’t part of Anthropic’s CVP program and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict.

“If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said. 

Chris Thompson — chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event — said that in his experience using the frontier AI models, the guardrails can be inconsistent and work differently every day. That’s true even inside the looser boundaries of Anthropic’s and OpenAI’s vetted programs. 

“I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” said Thompson. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.” 

Consequently, researchers rely on or get pushed toward Chinese open source models like GLM — freely downloadable models that can be run locally with no vetting or usage restrictions — said Thompson.

“You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

Rather than tightening restrictions further, Thompson called for the AI frontier labs to open up their programs, provide responsible access, and hold those who abuse their tools accountable. Otherwise, he argued, defenders will lose the AI race.

“There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” said Thompson. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.”

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

#guardrails #impeding #work #offensive #cybersecurity #researchers #TechCrunchcybersecurity,Zero-days

Post Comment