Never Use ChatGPT For These Five Tasks

The roster of people who have self-pwned by typing the wrong information into ChatGPT continues to grow. Samsung engineers, the acting director of America’s cyber defense agency, self-represented litigants who provide citations to court cases that don’t exist, a man who swapped table salt for sodium bromide after asking how to cut salt from his diet… the list is too long.

Even everyone’s favorite all-lowercase techbro CEO Sam Altman concedes that what you’re sharing with ChatGPT isn’t privileged. Speaking to podcaster Theo Von in July 2025, he highlighted how courts could force OpenAI to hand over chats, calling it “very screwed up.”

Tell a doctor, lawyer or therapist something and the law shields that conversation, but telling ChatGPT creates an audit trail. There’s a non-zero risk here, given that ChatGPT chats have been subpoenaed, exposed by a share button, indexed by Google and preserved by a court order post-deletion — all in the last 18 months.

Sure, the chatbot is arguably a useful tool for research (check that case citations are real), coding (even the bugs ship), explaining concepts (there’s no correlation between confidence and accuracy) and drafting (delete the prompt from your email before hitting send). But the areas below are where more than 900 million weekly users could create problems for themselves.

So to that end, don’t use ChatGPT (or any other AI chatbot)…

Law and data science researcher Damien Charlotin maintains a public database that, to date, has logged more than 2,000 court decisions worldwide hamstrung by ChatGPT. These are cases where a judge has caught AI-fabricated citations, quotes or arguments in a filing.

It’s not just self-representing litigants here. While they top the list with 1,175 records at the time of writing, lawyers come second with 815 records. It seems judges are growing tired of these shenanigans, with courts more recently imposing large fines well into the thousands-of-dollars range in a growing number of matters. Although the US makes up the largest number of cases in this data, judges in the UK, South Africa, Israel, Australia and Spain have all caught the same behavior. Where a tool was identified in those cases, ChatGPT was named more than any other.

OpenAI addressed this issue last October when it updated ChatGPT’s usage policies. Those policies now prohibit using its services for “tailored advice that requires a license” unless a licensed professional is involved. Beyond false citations, ChatGPT chats have no legal privilege, so anything typed into it about a live dispute is fair game for the other side’s lawyers in discovery. Go ahead and ask general questions like what a tort is (great if you need help falling asleep), but save the detailed questions about your own dispute for an actual lawyer.

OpenAI estimated last year that 0.15 percent of ChatGPT’s weekly active users have conversations that include explicit indicators of potential suicide planning. That unfortunately works out to more than a million people per week, with hundreds of thousands more users exhibiting signs of psychosis or mania. Altman has said young people in particular treat ChatGPT as a therapist or life coach and, again, that nothing shields these chats.

The parents of 16-year-old Adam Raine sued OpenAI for wrongful death in August 2025, alleging that ChatGPT reinforced their son’s suicidal ideation before his death. Since then, OpenAI has reworked its crisis responses with input from more than 170 clinicians. The company has also added parental controls to ChatGPT, though child safety experts are skeptical of ChatGPT for Teens, the age-gated version launched alongside those changes.

Researchers have found that chatbots tend to reinforce a user’s existing beliefs through sycophantic responses, in some cases feeding delusional thinking rather than challenging it. Describe an argument with your spouse, for example, and the model will work from your account’s context alone, probably serving you an affirming verdict back regardless. Legislators are beginning to wake up to this, with states including Illinois, Nevada and Maine having passed laws prohibiting AI from delivering therapy or making therapeutic decisions.

It should go without saying that uploading confidential work documents to a third-party chatbot is always a bad idea. Unfortunately, too many people don’t seem to realize this.

Ironically, Madhu Gottumukkala, the former acting director of the Cybersecurity and Infrastructure Security Agency, didn’t. He uploaded at least four contracting documents marked “for official use only” to the public version of ChatGPT last summer, triggering security alerts and a Department of Homeland Security review. Even more ironically, the tool had been blocked for most other DHS employees at the time; Gottumukkala requested a special exception to use it. Senator Chuck Grassley followed up in February this year demanding an account of what was exposed, but predictably, nobody was really held to account. Gottumukkala was eventually shuffled elsewhere within the DHS.

Samsung banned gen AI tools on company devices back in May 2023 after engineers notably pasted internal source code into ChatGPT. Accompanying that ban was an internal memo reasoning that data sent to such platforms lives on external servers where it’s difficult to retrieve or delete, and could end up disclosed to other users. On consumer accounts, chats can also be used to train new models unless you’ve chosen to opt out. But opted-out data is still stored and breachable, and subject to legal process. Pasting client data or anything subject to employer controls or NDAs into a personal ChatGPT discussion is almost always going to be a fireable offense; Samsung’s memo threatened exactly that.

The same October 2025 policy update that bans users from using ChatGPT as a lawyer also restricted people from seeking medical advice via the chatbot. HIPAA binds doctors, insurers and their business partners, but not OpenAI. Thus, health data that’s legally shielded in a patient portal or between a doctor and patient has no such protection if it’s disclosed to ChatGPT.

Researchers from Northeastern University reported in July that while OpenAI has tightened ChatGPT’s handling of suicide and self-harm, those safeguards largely don’t extend to other conditions. Their work found that conversations involving conditions like eating disorders, insomnia, substance abuse and bipolar disorder didn’t trigger the safeguards, and that hiding intent made every tested model’s protections more likely to fail.

In a particularly wacky case of ChatGPT medical advice gone bad, a 60-year-old man spent three weeks in the hospital last year after asking the chatbot how to cut chloride from his diet. He ended up convinced that sodium bromide, a toxic substance with a long half-life, was a safe swap for table salt. He bought it online, took it for three months and wound up in an emergency department convinced his neighbor had poisoned him. While the clinicians in this man’s case never saw his chat logs, ChatGPT offered bromide as a chloride substitution when they gave the same question to the bot themselves.

Fast Company found nearly 4,500 shared ChatGPT conversations lurking in Google search results last July. These included exchanges describing addiction, abuse and mental health struggles in enough detail to potentially identify their authors. The chats wound up indexed because people checked a “make this chat discoverable” checkbox in the share menu which, apparently, happened without them understanding what it did. OpenAI was quick to respond, killing the feature within a day of the report and purging conversations from Google.

As part of The New York Times’ copyright lawsuit, OpenAI was ordered in May 2025 to retain every consumer chat log, deleted and temporary conversations included. That obligation ran until September 26, 2025, and logs preserved during that window remained accessible to the plaintiffs. A judge later terminated that order, meaning OpenAI no longer has to preserve all its ChatGPT data. But for four months, it did, and it could happen again.

It’s vital to be aware that ChatGPT remembers what you tell it by default. That significantly raises the stakes for sensitive matters like passwords and account details, since they can persist across sessions and resurface in later conversations if mentioned even once. Passwords, one-time login codes, Social Security and passport numbers, full card or bank account details and photos of ID documents should never go into a prompt. Every one of those puts you at danger of identity theft if a chat is breached or otherwise accessed.

Altman has called for an “AI privilege” (of course he has) to fix these confidentiality issues. But until lawmakers create one, the safe assumption is that anything typed into ChatGPT could someday be read back to you by someone else. If you aren’t comfortable with that, don’t put it into a prompt.