This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

Wish you more bang for your buck

By ‘Cost a Bomb’ I didn’t mean to exaggerate. It’s an idiom that means ‘To be very expensive’. In fact, no app on the planet costs a bomb lol.

So, to the point.

Chronicle

(Freemium. For Pro, $24.99 | Available on Setapp)

You probably have trouble keeping track of dozens of bills and subscriptions each month.

Try Chronicle.

Easy-to-use app

• Add payment due dates and receive reminders,

• Save payment documentation,

• Analyze your spending by season, year, and month.

• Observe expenditure trends and create new budgets.

Best of all, Chronicle features an integrated browser for fast payment and logging.

iOS and macOS sync.

SoundSource

($39 for lifetime)

Background Music, a free macOS program, was featured in #6 of this post last month.

It controls per-app volume, stereo balance, and audio over its max level.

Background Music is fully supported. Additionally,

• Connect various speakers to various apps (Wow! ),

• change the audio sample rate for each app,

• To facilitate access, add a floating SoundSource window.

• Use its blocks in Shortcuts app,

• On the menu bar, include meters for output/input devices and running programs.

PixelSnap

($39 for lifetime | Available on Setapp)

This software is heaven for UI designers.

It aids you.

• quickly calculate screen distances (in pixels) ,

• Drag an area around an object to determine its borders,

• Measure the distances between the additional guides,

• screenshots should be pixel-perfect.

What’s more.

You can

• Adapt your tolerance for items with poor contrast and shadows.

• Use your Touch Bar to perform important tasks, if you have one.

Mate Translation

($3.99 a month / $29.99 a year | Available on Setapp)

Mate Translate resembles a roided-up version of BarTranslate, which I wrote about in #1 of this piece last month.

If you translate often, utilize Mate Translate on macOS and Safari.

I'm really vocal about it.

It stays on the menu bar, and is accessible with a click or ⌥+shift+T hotkey.

It lets you

• Translate in 103 different languages,

• To translate text, double-click or right-click on it.

• Totally translate websites. Additionally, Netflix subtitles,

• Listen to their pronunciation to see how close it is to human.

iPhone and Mac sync Mate-ing history.

Swish

($16 for lifetime | Available on Setapp)

Swish is awesome!

Swipe, squeeze, tap, and hold movements organize chaotic desktop windows. Swish operates with mouse and trackpad.

Some gestures:

• Pinch Once: Close an app
• Pinch Twice: Quit an app
• Swipe down once: Minimise an app
• Pinch Out: Enter fullscreen mode
• Tap, Hold, & Swipe: Arrange apps in grids
and many more...

After getting acquainted to the movements, your multitasking will improve.

Unite

($24.99 for lifetime | Available on Setapp)

It turns webapps into macOS apps. The end.

Unite's functionality is a million times better.

• Provide extensive customization (incl. its icon, light and dark modes)

• make menu bar applications,

• Get badges for web notifications and automatically refresh websites,

• Replace any dock icon in the window with it (Wow!) by selecting that portion of the window.

• Use PiP (Picture-in-Picture) on video sites that support it.

• Delete advertising,

• Throughout macOS, use floating windows

and many more…

I feel $24.99 one-off for this tool is a great deal, considering all these features. What do you think?

https://www.bzgapps.com/unite

CleanShot X

(Basic: $29 one-off. Pro: $8/month | Available on Setapp)

CleanShot X can achieve things the macOS screenshot tool cannot. Complete screenshot toolkit.

CleanShot X, like Pixel Snap 2 (#3), is fantastic.

Allows

• Scroll to capture a long page,

• screen recording,

  With webcam on,
• With mic and system audio,
• Highlighting mouse clicks and hotkeys.

• Maintain floating screenshots for reference

• While capturing, conceal desktop icons and notifications.

• Recognize text in screenshots (OCR),

• You may upload and share screenshots using the built-in cloud.

These are just 6 in 50+ features, and you’re already saying Wow!

We may never tap again in ten years.

I discussed styling websites and web apps on smartwatches in my earlier article on W3C standardization.

The Parallax Chronicles

Section containing examples and flying objects

Another intriguing Working Draft I found applies to all devices, including smartphones.

These pages may have something intriguing. Take your time. Return after scrolling:

• Exhibit 1

• Exhibit 2

• Exhibit 3

What connects these three pages?

JustinWick at English Wikipedia • CC-BY-SA-3.0

Scroll-linked animation, commonly called parallax, is the effect.

WordPress theme developers' quick setup and low-code tools made the effect popular around 2014.

Parallax: Why Designers Love It

The chapter that your designer shouldn't read

Online video playback required searching, scrolling, and clicking ten years ago. Scroll and click four years ago.

Some video sites let you swipe to autoplay the next video from an endless list.

UI designers create scrollable pages and apps to accommodate the behavioral change.

Web interactivity used to be mouse-based. Clicking a button opened a help drawer, and hovering animated it.

However, a large page with more material requires fewer buttons and less interactiveness.

Designers choose scroll-based effects. Design and frontend developers must fight the trend but prepare for the worst.

How to Create Parallax

The component that you might want to show the designer

JavaScript-based effects track page scrolling and apply animations.

Javascript libraries like lax.js simplify it.

Using it needs a lot of human mathematical and physical computations.

Your asset library must also be prepared to display your website on a laptop, television, smartphone, tablet, foldable smartphone, and possibly even a microwave.

Overall, scroll-based animations can be solved better.

CSS Scroll-linked Animations

CSS makes sense since it's presentational. A Working Draft has been laying the groundwork for the next generation of interactiveness.

The new CSS property scroll-timeline powers the feature, which MDN describes well.

Before testing it, you should realize it is poorly supported:

Firefox 103 currently supports it.

There is also a polyfill, with some demo examples to explore.

Summary

Web design was a protracted process. Started with pages with static backdrop images and scrollable text. Artists and designers may use the scroll-based animation CSS API to completely revamp our web experience.

It's a promising frontier. This post may attract a future scrollable web designer.

Ps. I have created flashcards for HTML, Javascript etc. Check them out!

I've spent over 50 hours researching one-person firms, which interest me. I've found countless one-person enterprises that made millions on the founder's determination and perseverance.

Throughout my investigation, I found three of the most outstanding one-person enterprises. These enterprises show that people who work hard and dedicate themselves to their ideas may succeed.

Eric Barone (@ConcernedApe) created Stardew Valley in 2011 to better his job prospects. Eric loved making the game, in which players inherit a farm, grow crops, raise livestock, make friends with the villagers, and form a family.

Eric handled complete game production, including 3D graphics, animations, and music, to maintain creative control. He stopped job hunting and worked 8-15 hours a day on the game.

Eric developed a Stardew Valley website and subreddit to engage with gamers and get feedback. Eric's devoted community helped him meet Steam's minimum vote requirement for single creators.

Stardew Valley sold 1 million copies in two months after Eric launched it for $15 in 2016. The game has sold 20 million copies and made $300 million.

The game's inexpensive price, outsourcing of PR, marketing, and publication, and loyal player base helped it succeed. Eric has turned down million-dollar proposals from Sony and Nintendo to sell the game and instead updates and improves it. Haunted Chocolatier is Eric's new game.

Is farming not profitable? Ask Stardew Valley creator Eric Barone.

Gary Brewer established BuiltWith to assist users find website technologies and services. BuiltWith boasts 3000 paying customers and $14 million in yearly revenue, making it a significant resource for businesses wishing to generate leads, do customer analytics, obtain business insight, compare websites, or search websites by keyword.

BuiltWith has one full-time employee, Gary, and one or two part-time contractors that help with the blog. Gary handles sales, customer service, and other company functions alone.

BuiltWith acquired popularity through blog promotions and a top Digg ranking. About Us, a domain directory, connected to BuiltWith on every domain page, boosting it. Gary introduced $295–$995 monthly subscriptions to search technology, keywords, and potential consumers in response to customer demand.

Gary uses numerous methods to manage a firm without staff. He spends one to two hours every day answering user queries, most of which are handled quickly by linking to BuiltWiths knowledge store. Gary creates step-by-step essays or videos for complex problems. Gary can focus on providing new features based on customer comments and requests since he makes it easy to unsubscribe.

BuiltWith is entirely automated and successful due to its unique approach and useful offerings. It works for Google, Meta, Amazon, and Twitter.

Digital Inspiration develops Google Documents, Sheets, and Slides plugins. Digital Inspiration, founded by Amit Agarwal, receives 5 million monthly visits and earns $10 million. 40 million individuals have downloaded Digital Inspirations plugins.

Amit started Digital Inspiration by advertising his blog at tech events and getting Indian filter blogs and other newspapers to promote his articles. Amit built plugins and promoted them on the blog once the blog acquired popularity, using ideas from comments, friends, and Reddit. Digital Inspiration has over 20 free and premium plugins.

Mail Merge, Notifications for Google Forms, YouTube Uploader, and Document Studio are some of Digital Inspiration's most popular plugins. Mail Merge allows users to send personalized emails in bulk and track email opens and clicks.

Since Amits manages Digital Inspiration alone, his success is astounding. Amit developed a successful company via hard work and creativity, despite platform dependence. His tale inspires entrepreneurs.