This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

TOP APPS

Who survives spring cleaning?

Relax. Notion is off-limits. This topic is popular.

(I wrote about it 2 years ago, before everyone else did.) So).

These apps are probably new to you. I hope you find a new phone app after reading this.

Outdooractive

ViewRanger is Google Maps for outdoor enthusiasts.

This app has been so important to me as a freedom-loving long-distance walker and hiker.

This app shows nearby trails and right-of-ways on top of an Open Street Map.

Helpful detail and data. Any route's distance,

You can download and follow tons of routes planned by app users.

This has helped me find new routes and places a fellow explorer has tried.

Free with non-intrusive ads. Years passed before I subscribed. Pro costs £2.23/month.

This app is for outdoor lovers.

Google Files

New phones come with bloatware. These rushed apps are frustrating.

We must replace these apps. 2017 was Google's year.

Files is a file manager. It's quick, innovative, and clean. They've given people what they want.

It's easy to organize files, clear space, and clear cache.

I recommend Gallery by Google as a gallery app alternative. It's quick and easy.

Trainline

App for trains, buses, and coaches.

I've used this app for years. It did the basics well when I first used it.

Since then, it's improved. It's constantly adding features to make traveling easier and less stressful.

Split-ticketing helps me save hundreds a year on train fares. This app is only available in the UK and Europe.

This service doesn't link to a third-party site. Their app handles everything.

Not all train and coach companies use this app. All the big names are there, though.

Here's more on the app.

Battlefield: Mobile

Play Store has 478,000 games. Few can turn my phone into a console.

Call of Duty Mobile and Asphalt 8/9 are examples.

Asphalt's loot boxes and ads make it unplayable. Call of Duty opens with a few ads. Close them to play without hassle.

This game uses all your phone's features to provide a high-quality, seamless experience. If my internet connection is good, I never experience lag or glitches.

The gameplay is energizing and intense, just like on consoles. Sometimes I'm too involved. I've thrown my phone in anger. I'm totally absorbed.

Customizability is my favorite. Since phones have limited screen space, we should only have the buttons we need, placed conveniently.

Size, opacity, and position are modifiable. Adjust audio, graphics, and textures. It's customizable.

This game has been on my phone for three years. It began well and has gotten better. When I think the creators can't do more, they do.

If you play, read my tips for winning a Battle Royale.

Lightroom

As a photographer, I believe your best camera is on you. The phone.

2017 was a big year for this app. I've tried many photo-editing apps since then. This always wins.

The app is dull. I've never seen better photo editing on a phone.

Adjusting settings and sliders doesn't damage or compress photos. It's detailed.

This is important for phone photos, which are lower quality than professional ones.

Some tools are behind a £4.49/month paywall. Adobe must charge a subscription fee instead of selling licenses. (I'm still bitter about Creative Cloud's price)

Snapseed is my pick. Lightroom is where I do basic editing before moving to Snapseed. Snapseed review:

These apps are great. They cover basic and complex editing needs while traveling.

Final Reflections

I hope you downloaded one of these. Share your favorite apps. These apps are scarce.

Here's how to be clear.

“I have only made this letter longer because I have not had the time to make it shorter.” — French mathematician, physicist, inventor, philosopher, and writer Blaise Pascal

Concise.

People want this. We tend to repeat ourselves and use unnecessary words.

Being vague frustrates readers. It focuses their limited attention span on figuring out what you're saying rather than your message.

Edit carefully.

“Examine every word you put on paper. You’ll find a surprising number that don’t serve any purpose.” — American writer, editor, literary critic, and teacher William Zinsser

How do you write succinctly?

Here are three ways to polish your writing.

1. Delete

Your readers will appreciate it if you delete unnecessary words. If a word or phrase is essential, keep it. Don't force it.

Many readers dislike bloated sentences. Ask yourself if cutting a word or phrase will change the meaning or dilute your message.

For example, you could say, “It’s absolutely essential that I attend this meeting today, so I know the final outcome.” It’s better to say, “It’s critical I attend the meeting today, so I know the results.”

Key takeaway

Delete actually, completely, just, full, kind of, really, and totally. Keep the necessary words, cut the rest.

2. Just Do It

Don't tell readers your plans. Your readers don't need to know your plans. Who are you?

Don't say, "I want to highlight our marketing's problems." Our marketing issues are A, B, and C. This cuts 5–7 words per sentence.

Keep your reader's attention on the essentials, not the fluff. What are you doing? You won't lose readers because you get to the point quickly and don't build up.

Key takeaway

Delete words that don't add to your message. Do something, don't tell readers you will.

3. Cut Overlap

You probably repeat yourself unintentionally. You may add redundant sentences when brainstorming. Read aloud to detect overlap.

Remove repetition from your writing. It's important to edit our writing and thinking to avoid repetition.

Key Takeaway

If you're repeating yourself, combine sentences to avoid overlap.

4. Simplify

Write as you would to family or friends. Communicate clearly. Don't use jargon. These words confuse readers.

Readers want specifics, not jargon. Write simply. Done.

Most adults read at 8th-grade level. Jargon and buzzwords make speech fluffy. This confuses readers who want simple language.

Key takeaway

Ensure all audiences can understand you. USA Today's 5th-grade reading level is intentional. They want everyone to understand.

5. Active voice

Subjects perform actions in active voice. When you write in passive voice, the subject receives the action.

For example, “the board of directors decided to vote on the topic” is an active voice, while “a decision to vote on the topic was made by the board of directors” is a passive voice.

Key takeaway

Active voice clarifies sentences. Active voice is simple and concise.

Bringing It All Together

Five tips help you write clearly. Delete, just do it, cut overlap, use simple language, and write in an active voice.

Clear writing is effective. It's okay to occasionally use unnecessary words or phrases. Realizing it is key. Check your writing.

Adding words costs.

Write more concisely. People will appreciate it and read your future articles, emails, and messages. Spending extra time will increase trust and influence.

“Not that the story need be long, but it will take a long while to make it short.” — Naturalist, essayist, poet, and philosopher Henry David Thoreau

“Poor distribution, not poor goods, is the main reason for failure” — Peter Thiel.

Here's an easy way to conceptualize "go-to-market" for your distribution plan.

One equation captures the concept:

Distribution = Ecosystem Participants + Incentives

Draw your customers' ecosystem. Set aside your goods and consider your consumer's environment. Who do they deal with daily? 

1. First, list each participant. You want an exhaustive list, but here are some broad categories.

• In-person media services

• Websites

• Events\Networks

• Financial education and banking

• Shops

• Staff

• Advertisers

• Twitter influencers

1. Draw influence arrows. Who's affected? I'm not just talking about Instagram selfie-posters. Who has access to your consumer and could promote your product if motivated?

The thicker the arrow, the stronger the relationship. Include more "influencers" if needed. Customer ecosystems are complex.

3. Incentivize ecosystem players. “Show me the incentive and I will show you the result.“, says Warren Buffet's business partner Charlie Munger.

Strong distribution strategies encourage others to promote your product to your target market by incentivizing the most prominent players. Incentives can be financial or non-financial.

Financial rewards

Usually, there's money. If you pay Facebook, they'll run your ad. Salespeople close deals for commission. Giving customers bonus credits will encourage referrals.

Most businesses underuse non-financial incentives.

Non-cash incentives

Motivate key influencers without spending money to expand quickly and cheaply. What can you give a client-connector for free?

Here are some ideas:

Are there any other features or services available?

Titles or status? Tinder paid college "ambassadors" for parties to promote its dating service.

Can I get early/free access? Facebook gave a select group of developers "exclusive" early access to their AR platform.

Are you a good host? Pharell performed at YPlan's New York launch party.

Distribution? Apple's iPod earphones are white so others can see them.

Have an interesting story? PR rewards journalists by giving them a compelling story to boost page views.

Prioritize distribution.

More time spent on distribution means more room in your product design and business plan. Once you've identified the key players in your customer's ecosystem, talk to them.

Money isn't your only resource. Creative non-monetary incentives may be more effective and scalable. Give people something useful and easy to deliver.

Trigger warning — Andrew Tate is running a genius marketing campaign

Andrew Tate is a 2022 internet celebrity.

Kickboxing world champion became rich playboy with controversial views on gender roles.

Andrew's get-rich-quick scheme isn't new. His social media popularity is impressive.

He’s currently running one of the most genius marketing campaigns in history.

He pulls society's pendulum away from diversity and inclusion and toward diversion and exclusion. He's unstoppable.

Here’s everything you need to know about Andrew Tate. And how he’s playing chess while the world plays checkers.

Cobra Tate is the name he goes by.

American-born, English-raised entrepreneur Andrew Tate lives in Romania.

Romania? Says Andrew,

“I prefer a country in which corruption is available to everyone.”

Andrew was a professional kickboxer with the ring moniker Cobra before starting Hustlers University.

Before that, he liked chess and worshipped his father.

Emory Andrew Tate III is named after his grandmaster chess player father.

Emory was the first black-American chess champion. He was military, martial arts-trained, and multilingual. A superhuman.

He lived in his car to make ends meet.

Andrew and Tristan relocated to England with their mother when their parents split.

It was there that Andrew began his climb toward becoming one of the internet’s greatest villains.

Andrew fell in love with kickboxing.

Andrew spent his 20s as a professional kickboxer and reality TV star, featuring on Big Brother UK and The Ultimate Traveller.

These 3 incidents, along with a chip on his shoulder, foreshadowed Andrews' social media breakthrough.

• Chess

• Combat sports

• Reality television

A dangerous trio.

Andrew started making money online after quitting kickboxing in 2017 due to an eye issue.

Andrew didn't suddenly become popular.

Andrew's web work started going viral in 2022.

Due to his contentious views on patriarchy and gender norms, he's labeled the King of Toxic Masculinity. His most contentious views (trigger warning):

• “Women are intrinsically lazy.”

• “Female promiscuity is disgusting.”

• “Women shouldn’t drive cars or fly planes.”

• “A lot of the world’s problems would be solved if women had their body count tattooed on their foreheads.”

Andrew's two main beliefs are:

1. “These are my personal opinions based on my experiences.”

2. “I believe men are better at some things and women are better at some things. We are not equal.”

Andrew intentionally offends.

Andrew's thoughts began circulating online in 2022.

In July 2022, he was one of the most Googled humans, surpassing:

• Joe Biden

• Donald Trump

• Kim Kardashian

Andrews' rise is a mystery since no one can censure or suppress him. This is largely because Andrew nor his team post his clips.

But more on that later.

Andrew's path to wealth.

Andrew Tate is a self-made millionaire. His morality is uncertain.

Andrew and Tristan needed money soon after retiring from kickboxing.

“I owed some money to some dangerous people. I had $70K and needed $100K to stay alive.”

Andrews lost $20K on roulette at a local casino.

Andrew had one week to make $50,000, so he started planning. Andrew locked himself in a chamber like Thomas Edison to solve an energy dilemma.

He listed his assets.

• Physical strength (but couldn’t fight)

• a BMW (worth around $20K)

• Intelligence (but no outlet)

A lightbulb.

He had an epiphany after viewing a webcam ad. He sought aid from women, ironically. His 5 international girlfriends are assets.

Then, a lightbulb.

Andrew and Tristan messaged and flew 7 women to a posh restaurant. Selling desperation masked as opportunity, Andrew pitched his master plan:

A webcam business — with a 50/50 revenue split.

5 women left.

2 stayed.

Andrew Tate, a broke kickboxer, became Top G, Cobra Tate.

The business model was simple — yet sad.

Andrew's girlfriends moved in with him and spoke online for 15+ hours a day. Andrew handled ads and equipment as the women posed.

Andrew eventually took over their keyboards, believing he knew what men wanted more than women.

Andrew detailed on the Full Send Podcast how he emotionally manipulated men for millions. They sold houses, automobiles, and life savings to fuel their companionship addiction.

When asked if he felt bad, Andrew said,

“F*ck no.“

Andrew and Tristan wiped off debts, hired workers, and diversified.

Tristan supervised OnlyFans models.

Andrew bought Romanian casinos and MMA league RXF (Real Xtreme Fighting).

Pandemic struck suddenly.

Andrew couldn't run his 2 businesses without a plan. Another easy moneymaker.

He banked on Hustlers University.

The actual cause of Andrew's ubiquity.

On a Your Mom’s House episode Andrew's 4 main revenue sources:

1. Hustler’s University

2. Owning casinos in Romania

3. Owning 10% of the Romanian MMA league “RXF”

4. “The War Room” — a society of rich and powerful men

When the pandemic hit, 3/4 became inoperable.

So he expanded Hustlers University.

But what is Hustler’s University?

Andrew says Hustlers University teaches 18 wealth-building tactics online. Examples:

• Real estate

• Copywriting

• Amazon FBA

• Dropshipping

• Flipping Cryptos

How to swiftly become wealthy.

Lessons are imprecise, rudimentary, and macro-focused, say reviews. Invest wisely, etc. Everything is free online.

You pay for community. One unique income stream.

The only money-making mechanism that keeps the course from being a scam.

The truth is, many of Andrew’s students are actually making money. Maybe not from the free YouTube knowledge Andrew and his professors teach in the course, but through Hustler’s University’s affiliate program.

Affiliates earn 10% commission for each new student = $5.

Students can earn $10 for each new referral in the first two months.

Andrew earns $50 per membership per month.

This affiliate program isn’t anything special — in fact, it’s on the lower end of affiliate payouts. Normally, it wouldn’t be very lucrative.

But it has one secret weapon— Andrew and his viral opinions.

Andrew is viral. Andrew went on a media tour in January 2022 after appearing on Your Mom's House.

• The Full Send Podcast

• The Fresh & Fit Podcast

• “BFFs” by Barstool Sports

And many, many more…

He chatted with Twitch streamers. Hustlers University wanted more controversy (and clips).

Here’s the strategy behind Hustler’s University that has (allegedly) earned students upwards of $10K per month:

1. Make a social media profile with Andrew Tates' name and photo.

2. Post any of the online videos of Andrews that have gone viral.

3. Include a referral link in your bio.

Effectively simple.

Andrew's controversy attracts additional students. More student clips circulate as more join. Andrew's students earn more and promote the product as he goes viral.

A brilliant plan that's functioning.

At the beginning of his media tour, Hustler’s University had 5,000 students. 6 months in, and he now has over 100,000.

One income stream generates $5 million every month.

Andrew's approach is not new.

But it is different.

In the early 2010s, Tai Lopez dominated the internet.

His viral video showed his house.

“Here in my garage. Just bought this new Lamborghini.”

Tais' marketing focused on intellect, not strength, power, and wealth to attract women.

How reading quicker leads to financial freedom in 67 steps.

Years later, it was revealed that Tai Lopez rented the mansion and Lamborghini as a marketing ploy to build social proof. Meanwhile, he was living in his friend’s trailer.

Faked success is an old tactic.

Andrew is doing something similar. But with one major distinction.

Andrew outsources his virality — making him nearly impossible to cancel.

In 2022, authorities searched Andrews' estate over human trafficking suspicions. Investigation continues despite withdrawn charges.

Andrew's divisive nature would normally get him fired. Andrew's enterprises and celebrity don't rely on social media.

He doesn't promote or pay for ads. Instead, he encourages his students and anyone wishing to get rich quick to advertise his work.

Because everything goes through his affiliate program. Old saying:

“All publicity is good publicity.”

Final thoughts: it’s ok to feel triggered.

Tate is divisive.

His emotionally charged words are human nature. Andrews created the controversy.

It's non-personal.

His opinions are those of one person. Not world nor generational opinion.

Briefly:

• It's easy to understand why Andrews' face is ubiquitous. Money.

• The world wide web is a chessboard. Misdirection is part of it.

• It’s not personal, it’s business.

• Controversy sells

Sometimes understanding the ‘why’, can help you deal with the ‘what.’