This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

Dark Web activity? Is it as horrible as they say?

We peruse our phones for hours. Internet has improved our worldview.

However, the world's harshest realities remain buried on the internet and unattainable by everyone.

Browsers cannot access the Dark Web. Browse it with high-security authentication and exclusive access. There are compelling reasons to avoid the dark web at all costs.

1. The Dark Web and I

Darius wrote My Dark Web Story on reddit two years ago. The user claimed to have shared his dark web experience. DaRealEddyYT wanted to surf the dark web after hearing several stories.

He curiously downloaded Tor Browser, which provides anonymity and security.

In the Dark Room, bound

As Darius logged in, a text popped up: “Want a surprise? Click on this link.”

The link opened to a room with a chair. Only one light source illuminated the room. The chair held a female tied.

As the screen read "Let the game begin," a man entered the room and was paid in bitcoins to torment the girl.

The man dragged and tortured the woman.

A danger to safety

Leaving so soon, Darius, disgusted Darius tried to leave the stream. The anonymous user then sent Darius his personal information, including his address, which frightened him because he didn't know Tor was insecure.

After deleting the app, his phone camera was compromised.

He also stated that he left his residence and returned to find it unlocked and a letter saying, Thought we wouldn't find you? Reddit never updated the story.

The story may have been a fake, but a much scarier true story about the dark side of the internet exists.

2. The Silk Road Market

The dark web is restricted for a reason. The dark web has everything illicit imaginable. It's awful central.

The dark web has everything, from organ sales to drug trafficking to money laundering to human trafficking. Illegal drugs, pirated software, credit card, bank, and personal information can be found in seconds.

The dark web has reserved websites like Google. The Silk Road Website, which operated from 2011 to 2013, was a leading digital black market.

The FBI grew obsessed with site founder and processor Ross William Ulbricht.

The site became a criminal organization as money laundering and black enterprises increased. Bitcoin was utilized for credit card payment.

The FBI was close to arresting the site's administrator. Ross was detained after the agency closed Silk Road in 2013.

Two years later, in 2015, he was convicted and sentenced to two consecutive life terms and forty years. He appealed in 2016 but was denied, thus he is currently serving time.

The hefty sentence was for more than running a black marketing site. He was also convicted of murder-for-hire, earning about $730,000 in a short time.

3. Person-buying auctions

Bidding on individuals is another weird internet activity. After a Milan photo shoot, 20-year-old British model Chloe Ayling was kidnapped.

An ad agency in Milan made a bogus offer to shoot with the mother of a two-year-old boy. Four men gave her anesthetic and put her in a duffel bag when she arrived.

She was held captive for several days, and her images and $300,000 price were posted on the dark web. Black Death Trafficking Group kidnapped her to sell her for sex.

She was told two black death foot warriors abducted her. The captors released her when they found she was a mother because mothers were less desirable to sex slave buyers.

In July 2018, Lukasz Pawel Herba was arrested and sentenced to 16 years and nine months in prison. Being a young mother saved Chloe from creepy bidding.

However, it exceeds expectations of how many more would be in such danger daily without their knowledge.

4. Organ sales

Many are unaware of dark web organ sales. Patients who cannot acquire organs often turn to dark web brokers.

Brokers handle all transactions between donors and customers.

Bitcoins are used for dark web transactions, and the Tor server permits personal data on the web.

The WHO reports approximately 10,000 unlawful organ transplants annually. The black web sells kidneys, hearts, even eyes.

To protect our lives and privacy, we should manage our curiosity and never look up dangerous stuff.

While it's fascinating and appealing to know what's going on in the world we don't know about, it's best to prioritize our well-being because one never knows how bad it might get.

Sources

Reddit.com

The Daily Beast

PYMNTS

Commons.erau.edu

The Sun

Investopedia

Startup Talky

It's cheaper, faster charging, longer lasting, safer, and better for the environment.

Batteries are the future. Next-gen and planet-saving technology, including solar power and EVs, require batteries. As these smart technologies become more popular, we find that our batteries can't keep up. Lithium-ion batteries are expensive, slow to charge, big, fast to decay, flammable, and not environmentally friendly. MIT just created a new battery that eliminates all of these problems.  So, is this the battery of the future? Or is there a catch?

When I say entirely new, I mean it. This battery employs no currently available materials. Its electrodes are constructed of aluminium and pure sulfur instead of lithium-complicated ion's metals and graphite. Its electrolyte is formed of molten chloro-aluminate salts, not an organic solution with lithium salts like lithium-ion batteries.

How does this change in materials help?

Aluminum, sulfur, and chloro-aluminate salts are abundant, easy to acquire, and cheap. This battery might be six times cheaper than a lithium-ion battery and use less hazardous mining. The world and our wallets will benefit.

But don’t go thinking this means it lacks performance.

This battery charged in under a minute in tests. At 25 degrees Celsius, the battery will charge 25 times slower than at 110 degrees Celsius. This is because the salt, which has a very low melting point, is in an ideal state at 110 degrees and can carry a charge incredibly quickly. Unlike lithium-ion, this battery self-heats when charging and discharging, therefore no external heating is needed.

Anyone who's seen a lithium-ion battery burst might be surprised. Unlike lithium-ion batteries, none of the components in this new battery can catch fire. Thus, high-temperature charging and discharging speeds pose no concern.

These batteries are long-lasting. Lithium-ion batteries don't last long, as any iPhone owner can attest. During charging, metal forms a dendrite on the electrode. This metal spike will keep growing until it reaches the other end of the battery, short-circuiting it. This is why phone batteries only last a few years and why electric car range decreases over time. This new battery's molten salt slows deposition, extending its life. This helps the environment and our wallets.

These batteries are also energy dense. Some lithium-ion batteries have 270 Wh/kg energy density (volume and mass). Aluminum-sulfur batteries could have 1392 Wh/kg, according to calculations. They'd be 5x more energy dense. Tesla's Model 3 battery would weigh 96 kg instead of 480 kg if this battery were used. This would improve the car's efficiency and handling.

These calculations were for batteries without molten salt electrolyte. Because they don't reflect the exact battery chemistry, they aren't a surefire prediction.

This battery seems great. It will take years, maybe decades, before it reaches the market and makes a difference. Right?

Nope. The project's scientists founded Avanti to develop and market this technology.

So we'll soon be driving cheap, durable, eco-friendly, lightweight, and ultra-safe EVs? Nope.

This battery must be kept hot to keep the salt molten; otherwise, it won't work and will expand and contract, causing damage. This issue could be solved by packs that can rapidly pre-heat, but that project is far off.

Rapid and constant charge-discharge cycles make these batteries ideal for solar farms, homes, and EV charging stations. The battery is constantly being charged or discharged, allowing it to self-heat and maintain an ideal temperature.

These batteries aren't as sexy as those making EVs faster, more efficient, and cheaper. Grid batteries are crucial to our net-zero transition because they allow us to use more low-carbon energy. As we move away from fossil fuels, we'll need millions of these batteries, so the fact that they're cheap, safe, long-lasting, and environmentally friendly will be huge. Who knows, maybe EVs will use this technology one day. MIT has created another world-changing technology.

Google announced 6% layoffs recently (or 12,000 people). This aligns it with most tech companies. A publicly contrite CEO explained that they had overhired during the COVID-19 pandemic boom and had to address it, but they were sorry and took full responsibility. I thought this was "bullshit" too. Meta, Amazon, Microsoft, and others must feel similarly. I spent 10 years at Google, and these things don't reflect well on the company's leaders.

All publicly listed companies have a fiduciary duty to act in the best interests of their shareholders. Dodge vs. Ford Motor Company established this (1919). Henry Ford wanted to reduce shareholder payments to offer cheaper cars and better wages. Ford stated.

My ambition is to employ still more men, to spread the benefits of this industrial system to the greatest possible number, to help them build up their lives and their homes. To do this we are putting the greatest share of our profits back in the business.

The Dodge brothers, who owned 10% of Ford, opposed this and sued Ford for the payments to start their own company. They won, preventing Ford from raising prices or salaries. If you have a vocal group of shareholders with the resources to sue you, you must prove you are acting in their best interests. Companies prioritize shareholders. Giving activist investors a stick to threaten you almost enshrines short-term profit over long-term thinking.

This underpins Google's current issues. Institutional investors who can sue Google see it as a wasteful company they can exploit. That doesn't mean you have to maximize profits (thanks to those who pointed out my ignorance of US corporate law in the comments and on HN), but it allows pressure. I feel for those navigating this. This is about unrestrained capitalism.

When Google went public, Larry Page and Sergey Brin knew the risks and worked hard to keep control. In their Founders' Letter to investors, they tried to set expectations for the company's operations.

Our long-term focus as a private company has paid off. Public companies do the same. We believe outside pressures lead companies to sacrifice long-term opportunities to meet quarterly market expectations.

The company has transformed since that letter. The company has nearly 200,000 full-time employees and a trillion-dollar market cap. Large investors have bought company stock because it has been a good long-term bet. Why are they restless now?

Other big tech companies emerged and fought for top talent. This has caused rising compensation packages. Google has also grown rapidly (roughly 22,000 people hired to the end of 2022). At $300,000 median compensation, those 22,000 people added $6.6 billion in salary overheads in 2022. Exorbitant. If the company still makes $16 billion every quarter, maybe not. Investors wonder if this value has returned.

Investors are right. Google uses people wastefully. However, by bluntly reducing headcount, they're not addressing the root causes and hurting themselves. No studies show that downsizing this way boosts productivity. There is plenty of evidence that they'll lose out because people will be risk-averse and distrust their leadership.

The company's approach also stinks. Finding out that you no longer have a job because you can’t log in anymore (sometimes in cases where someone is on call for protecting your production systems) is no way to fire anyone. Being with a narcissistic sociopath is like being abused. First, you receive praise and fancy perks for making the cut. You're fired by text and ghosted. You're told to appreciate the generous severance package. This firing will devastate managers and teams. This type of firing will take years to recover self-esteem. Senior management contributed to this. They chose the expedient answer, possibly by convincing themselves they were managing risk and taking the Macbeth approach of “If it were done when ’tis done, then ’twere well It were done quickly”.

Recap. Google's leadership did a stupid thing—mass firing—in a stupid way. How do we get rid of enough people to make investors happier? and "have 6% less people." Empathetic leaders should not emulate Elon Musk. There is no humane way to fire 12,000 people, but there are better ways. Why is Google so wasteful?

Ambition answers this. There aren't enough VP positions for a group of highly motivated, ambitious, and (increasingly) ruthless people. I’ve loitered around the edges of this world and a large part of my value was to insulate my teams from ever having to experience it. It’s like Game of Thrones played out through email and calendar and over video call.

Your company must look a certain way to be promoted to director or higher. You need the right people at the right levels under you. Long-term, growing your people will naturally happen if you're working on important things. This takes time, and you're never more than 6–18 months from a reorg that could start you over. Ambitious people also tend to be impatient. So, what do you do?

Hiring and vanity projects. To shape your company, you hire at the right levels. You value vanity metrics like active users over product utility. Your promo candidates get through by subverting the promotion process. In your quest for growth, you avoid performance managing people out. You avoid confronting toxic peers because you need their support for promotion. Your cargo cult gets you there.

Its ease makes Google wasteful. Since they don't face market forces, the employees don't see it as a business. Why would you do when the ads business is so profitable? Complacency causes senior leaders to prioritize their own interests. Empires collapse. Personal ambition often trumped doing the right thing for users, the business, or employees. Leadership's ambition over business is the root cause. Vanity metrics, mass hiring, and vague promises have promoted people to VP. Google goes above and beyond to protect senior leaders.

The decision-makers and beneficiaries are not the layoffees. Stock price increase beneficiaries. The people who will post on LinkedIn how it is about misjudging the market and how they’re so sorry and take full responsibility. While accumulating wealth, the dark room dwellers decide who stays and who goes. The billionaire investors. Google should start by addressing its bloated senior management, but — as they say — turkeys don't vote for Christmas. It should examine its wastefulness and make tough choices to fix it. A 6% cut is a blunt tool that admits you're not running your business properly. why aren’t the people running the business the ones shortly to be entering the job market?

This won't fix Google's wastefulness. The executives may never regain trust after their approach. Suppressed creativity. Business won't improve. Google will have lost its founding vision and us all. Large investors know they can force Google's CEO to yield. The rich will get richer and rationalize leaving 12,000 people behind. Cycles repeat.

It doesn’t have to be this way. In 2013, Nintendo's CEO said he wouldn't fire anyone for shareholders. Switch debuted in 2017. Nintendo's stock has increased by nearly five times, or 19% a year (including the drop most of the stock market experienced last year). Google wasted 12,000 talented people. To please rich people.

Don't say this in public!

Want to wow your audience? Want to deliver a successful presentation? Do you want practical takeaways from your presentation?

Then avoid these phrases.

Public speaking is difficult. People fear public speaking, according to research.

"Public speaking is people's biggest fear, according to studies. Number two is death. "Sounds right?" — Comedian Jerry Seinfeld

Yes, public speaking is scary. These words and phrases will make your presentation harder.

Using unnecessary words can weaken your message.

You may have prepared well for your presentation and feel confident. During your presentation, you may freeze up. You may blank or forget.

Effective delivery is even more important than skillful public speaking.

Here are 10 presentation pitfalls.

1. I or Me

Presentations are about the audience, not you. Replace "I or me" with "you, we, or us." Focus on your audience. Reward them with expertise and intriguing views about your issue.

Serve your audience actionable items during your presentation, and you'll do well. Your audience will have a harder time listening and engaging if you're self-centered.

2. Sorry if/for

Your presentation is fine. These phrases make you sound insecure and unprepared. Don't pressure the audience to tell you not to apologize. Your audience should focus on your presentation and essential messages.

3. Excuse the Eye Chart, or This slide's busy

Why add this slide if you're utilizing these phrases? If you don't like this slide, change it before presenting. After the presentation, extra data can be provided.

Don't apologize for unclear slides. Hide or delete a broken PowerPoint slide. If so, divide your message into multiple slides or remove the "business" slide.

4. Sorry I'm Nervous

Some think expressing yourself will win over the audience. Nerves are horrible. Even public speakers are nervous.

Nerves aren't noticeable. What's the point? Let the audience judge your nervousness. Please don't make this obvious.

5. I'm not a speaker or I've never done this before.

These phrases destroy credibility. People won't listen and will check their phones or computers.

Why present if you use these phrases?

Good speakers aren't necessarily public speakers. Be confident in what you say. When you're confident, many people will like your presentation.

6. Our Key Differentiators Are

Overused term. It's widely utilized. This seems "salesy," and your "important differentiators" are probably like a competitor's.

This statement has been diluted; say, "what makes us different is..."

7. Next Slide

Many slides or stories? Your presentation needs transitions. They help your viewers understand your argument.

You didn't transition well when you said "next slide." Think about organic transitions.

8. I Didn’t Have Enough Time, or I’m Running Out of Time

The phrase "I didn't have enough time" implies that you didn't care about your presentation. This shows the viewers you rushed and didn't care.

Saying "I'm out of time" shows poor time management. It means you didn't rehearse enough and plan your time well.

9. I've been asked to speak on

This phrase is used to emphasize your importance. This phrase conveys conceit.

When you say this sentence, you tell others you're intelligent, skilled, and appealing. Don't utilize this term; focus on your topic.

10. Moving On, or All I Have

These phrases don't consider your transitions or presentation's end. People recall a presentation's beginning and end.

How you end your discussion affects how people remember it. You must end your presentation strongly and use natural transitions.

Conclusion

10 phrases to avoid in a presentation. I or me, sorry if or sorry for, pardon the Eye Chart or this busy slide, forgive me if I appear worried, or I'm really nervous, and I'm not good at public speaking, I'm not a speaker, or I've never done this before.

Please don't use these phrases: next slide, I didn't have enough time, I've been asked to speak about, or that's all I have.

We shouldn't make public speaking more difficult than it is. We shouldn't exacerbate a difficult issue. Better public speakers avoid these words and phrases.

“Remember not only to say the right thing in the right place, but far more difficult still, to leave unsaid the wrong thing at the tempting moment.” — Benjamin Franklin, Founding Father

This is a summary. See the original post here.