This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

Wish you more bang for your buck

By ‘Cost a Bomb’ I didn’t mean to exaggerate. It’s an idiom that means ‘To be very expensive’. In fact, no app on the planet costs a bomb lol.

So, to the point.

Chronicle

(Freemium. For Pro, $24.99 | Available on Setapp)

You probably have trouble keeping track of dozens of bills and subscriptions each month.

Try Chronicle.

Easy-to-use app

• Add payment due dates and receive reminders,

• Save payment documentation,

• Analyze your spending by season, year, and month.

• Observe expenditure trends and create new budgets.

Best of all, Chronicle features an integrated browser for fast payment and logging.

iOS and macOS sync.

SoundSource

($39 for lifetime)

Background Music, a free macOS program, was featured in #6 of this post last month.

It controls per-app volume, stereo balance, and audio over its max level.

Background Music is fully supported. Additionally,

• Connect various speakers to various apps (Wow! ),

• change the audio sample rate for each app,

• To facilitate access, add a floating SoundSource window.

• Use its blocks in Shortcuts app,

• On the menu bar, include meters for output/input devices and running programs.

PixelSnap

($39 for lifetime | Available on Setapp)

This software is heaven for UI designers.

It aids you.

• quickly calculate screen distances (in pixels) ,

• Drag an area around an object to determine its borders,

• Measure the distances between the additional guides,

• screenshots should be pixel-perfect.

What’s more.

You can

• Adapt your tolerance for items with poor contrast and shadows.

• Use your Touch Bar to perform important tasks, if you have one.

Mate Translation

($3.99 a month / $29.99 a year | Available on Setapp)

Mate Translate resembles a roided-up version of BarTranslate, which I wrote about in #1 of this piece last month.

If you translate often, utilize Mate Translate on macOS and Safari.

I'm really vocal about it.

It stays on the menu bar, and is accessible with a click or ⌥+shift+T hotkey.

It lets you

• Translate in 103 different languages,

• To translate text, double-click or right-click on it.

• Totally translate websites. Additionally, Netflix subtitles,

• Listen to their pronunciation to see how close it is to human.

iPhone and Mac sync Mate-ing history.

Swish

($16 for lifetime | Available on Setapp)

Swish is awesome!

Swipe, squeeze, tap, and hold movements organize chaotic desktop windows. Swish operates with mouse and trackpad.

Some gestures:

• Pinch Once: Close an app
• Pinch Twice: Quit an app
• Swipe down once: Minimise an app
• Pinch Out: Enter fullscreen mode
• Tap, Hold, & Swipe: Arrange apps in grids
and many more...

After getting acquainted to the movements, your multitasking will improve.

Unite

($24.99 for lifetime | Available on Setapp)

It turns webapps into macOS apps. The end.

Unite's functionality is a million times better.

• Provide extensive customization (incl. its icon, light and dark modes)

• make menu bar applications,

• Get badges for web notifications and automatically refresh websites,

• Replace any dock icon in the window with it (Wow!) by selecting that portion of the window.

• Use PiP (Picture-in-Picture) on video sites that support it.

• Delete advertising,

• Throughout macOS, use floating windows

and many more…

I feel $24.99 one-off for this tool is a great deal, considering all these features. What do you think?

https://www.bzgapps.com/unite

CleanShot X

(Basic: $29 one-off. Pro: $8/month | Available on Setapp)

CleanShot X can achieve things the macOS screenshot tool cannot. Complete screenshot toolkit.

CleanShot X, like Pixel Snap 2 (#3), is fantastic.

Allows

• Scroll to capture a long page,

• screen recording,

  With webcam on,
• With mic and system audio,
• Highlighting mouse clicks and hotkeys.

• Maintain floating screenshots for reference

• While capturing, conceal desktop icons and notifications.

• Recognize text in screenshots (OCR),

• You may upload and share screenshots using the built-in cloud.

These are just 6 in 50+ features, and you’re already saying Wow!

They depend heavily on open source..

Discord users send billions of messages daily. Users wish to search these messages. How do we index these to search by message keywords?

Let’s find out.

1. Discord utilizes Elasticsearch. Elasticsearch is a free, open search engine for textual, numerical, geographical, structured, and unstructured data. Apache Lucene powers Elasticsearch.
   

2. How does elastic search store data? It stores it as numerous key-value pairs in JSON documents.
   

3. How does elastic search index? Elastic search's index is inverted. An inverted index lists every unique word in every page and where it appears.

4. Elasticsearch indexes documents and generates an inverted index to make data searchable in near real-time. The index API adds or updates JSON documents in a given index.

1. Let's examine how discord uses Elastic Search. Elasticsearch prefers bulk indexing. Discord couldn't index real-time messages. You can't search posted messages. You want outdated messages.

6. Let's check what bulk indexing requires.
1. A temporary queue for incoming communications.
2. Indexer workers that index messages into elastic search.

1. Discord's queue is Celery. The queue is open-source. Elastic search won't run on a single server. It's clustered. Where should a message go? Where?

8. A shard allocator decides where to put the message. Nevertheless. Shattered? A shard combines elastic search and index on. So, these two form a shard which is used as a unit by discord. The elastic search itself has some shards. But this is different, so don’t get confused.

1. Now, the final part is service discovery — to discover the elastic search clusters and the hosts within that cluster. This, they do with the help of etcd another open source tool.
   

A great thing to notice here is that discord relies heavily on open source systems and their base implementations which is very different from a lot of other products.

11 science-backed ways to improve health, happiness, healing, relaxation, and mental clarity.

Vagus nerve is the main parasympathetic nervous system component.

It helps us rest and digest by slowing and stabilizing a resting heart rate, slowing and stabilizing the breath, promoting digestion, improving recovery and healing times, producing saliva, releasing endorphins and hormones like dopamine, oxytocin, and serotonin, and boosting the immune, digestive, and cardiovascular systems.

The vagus nerve sends anti-inflammatory signals to other parts of the body and is located behind the tongue, in the throat, neck, heart, lungs, abdomen, and brainstem.

Vagus means wandering in Latin. So, it's bold.

Here are 11 proven ways to boost health, happiness, and the vagus nerve.

1. Extend

“Yoga stimulates different nerves in your body, especially the vagus nerve that carries information from the brain to most of the body’s major organs, slows everything down and allows self-regulation. It’s the nerve that is associated with the parasympathetic system and emotions like love, joy, and compassion.” — Deepak Chopra

Stretching doesn't require a yoga background.

Listen to your body and ease into simple poses. This connects the mind and body.

If you're new to yoga or don't have access to an in-person class, try Yoga with Adrienne. Over 600 YouTube videos give her plenty of material.

2. Inhale

Because inhaling and exhaling activate the autonomic nervous system, we can breathe to relax.

Exhaling activates the parasympathetic nervous system (rest and digest). One inhales stress, the other exhales it.

So, faster or more intense breathing increases stress. Slower breathing relaxes us.

Breathe slowly, smoothly, and less.

Rhythmic breathing helps me relax.

What to do is as follows:

1. Take 4 smooth, forceless nose breaths.

2. Exhale smoothly and forcefully for 4 seconds

3. Don't pause at the inhale or exhale.

4. Continue for 5 minutes/40 breaths

5. Hold your breath as long as comfortable.

6. Breathe normally.

If four seconds is too long, try breathing in and out for two seconds, or in and out for three seconds, until your breath naturally relaxes. Once calmer, extend your breath.

Any consistent rhythm without force is good. Your heart will follow your lead and become coherent.

3. Chant/Hum

Singing, chanting, or humming activate the vagus nerve through the back of the throat.

Humming emits nitric oxide.

Nitric oxide improves blood circulation, blood flow, heart health, and blood pressure.

Antiviral, antibacterial, anti-inflammatory, antioxidant, and antimicrobial properties kill viruses and bacteria in the nose and throat.

Gargling water stimulates the vagus nerve.

Simple ways to heal, boost energy, and boost mood are often the healthiest. They're free and can be done anywhere.

4. Have more fun

Laughing stimulates the throat muscles, activating the vagus nerve. What's not to like? It releases dopamine.

Take time to enjoy life. Maybe it's a book, podcast, movie, socializing with friends, or laughing yoga.

Follow your bliss, as Joseph Campbell says.

Laugh at yourself

Actually. Really.

Gagging activates vagus nerve-connected muscles. Some doctors use the gag reflex to test the vagus nerve.

Grossness isn't required. While brushing, gag quickly. My girlfriend's brother always does it.

I'm done brushing when I gag, he says.

6. Take in the outdoors

Nature relaxes body and mind. Better if you can walk barefoot.

Earthing is associated with hippies dancing in daisies.

Science now supports hippies.

7. Enter some chilly water.

The diving reflex activates the vagus nerve when exposed to cold water.

The diving reflex involves holding your breath in cold water. Cold showers work best.

Within minutes of being in cold water, parasympathetic nervous system activity, which calms the body, increases.

8. Workout

Exercise increases dopamine, blood circulation, and breathing. So we feel energized, calm, and well-rested.

After resting, the parasympathetic nervous system engages.

It's worth waiting for, though.

9. Play music with brainwaves

Brainwave music harmonizes brainwave activity, boosts productivity and mental clarity, and promotes peace and relaxation by stimulating the vagus nerve.

Simply play a song.

My favorite.

10. Make gentle eyes

Eyes, like breath, often reflect inner state. Sharp, dilated, focused eyes indicate alertness.

Soft, open eyes reflect relaxation and ease. Soft eyes relax the nervous system.

This practice reduces stress, anxiety, and body tension. It's a quick and effective way to enter a calm, peaceful state.

Wild animals can be hunted one minute and graze the next.

Put it into action:

Relax while seated.

Gaze at a distant object

Use peripheral vision while looking straight ahead

Without moving your eyes, look up and down. Connect side spaces to your vision.

Focus on everything as your eyes soften.

Keep breathing

Stay as long as you like

11. Be intimate

We kiss, moan, and breathe deeper during love. We get dopamine, oxytocin, serotonin, and vagus nerve stimulation.

Why not?

To sum up

Here are 11 vagus nerve resets:

1. Stretch

2. Breathe

3. Hum/Chant

4. More humor

5. Amuse yourself

6. Spend time outdoors

7. Leap into chilly water

8. Exercise

9. Play music with brainwaves.

10. Make gentle eyes.

11. Be intimate

If these words have inspired you, try my favorite breathwork technique. Combining breathing, chanting, and brainwave music. Win-win-win :)

Corn Kid got viral on TikTok after being interviewed by Recess Therapy. Tariq, called the Corn Kid, ate a buttery ear of corn in the video. He's corn crazy. He thinks everyone just has to try it. It turns out, whether we know it or not, we already have.

Corn is a fruit, veggie, and grain. It's the second-most-grown crop. Corn makes up 36% of U.S. exports. In the U.S., it's easy to grow and provides high yields, as proven by the vast corn belt spanning the Midwest, Great Plains, and Texas panhandle. Since 1950, the corn crop has doubled to 10 billion bushels.

You say, "Fine." We shouldn't just grow because we can. Why so much corn? What's this corn for?

Why is practical and political. Michael Pollan's The Omnivore's Dilemma has the full narrative. Early 1970s food costs increased. Nixon subsidized maize to feed the public. Monsanto genetically engineered corn seeds to make them hardier, and soon there was plenty of corn. Everyone ate. Woot! Too much corn followed. The powers-that-be had to decide what to do with leftover corn-on-the-cob.

They are fortunate that corn has a wide range of uses.

First, the edible variants. I divide corn into obvious and stealth.

Obvious corn includes popcorn, canned corn, and corn on the cob. This form isn't always digested and often comes out as entire, polka-dotting poop. Cornmeal can be ground to make cornbread, polenta, and corn tortillas. Corn provides antioxidants, minerals, and vitamins in moderation. Most synthetic Vitamin C comes from GMO maize.

Corn oil, corn starch, dextrose (a sugar), and high-fructose corn syrup are often overlooked. They're stealth corn because they sneak into practically everything. Corn oil is used for frying, baking, and in potato chips, mayonnaise, margarine, and salad dressing. Baby food, bread, cakes, antibiotics, canned vegetables, beverages, and even dairy and animal products include corn starch. Dextrose appears in almost all prepared foods, excluding those with high-fructose corn syrup. HFCS isn't as easily digested as sucrose (from cane sugar). It can also cause other ailments, which we'll discuss later.

Most foods contain corn. It's fed to almost all food animals. 96% of U.S. animal feed is corn. 39% of U.S. corn is fed to livestock. But animals prefer other foods. Omnivore chickens prefer insects, worms, grains, and grasses. Captive cows are fed a total mixed ration, which contains corn. These animals' products, like eggs and milk, are also corn-fed.

There are numerous non-edible by-products of corn that are employed in the production of items like:

1. fuel-grade ethanol

2. plastics

3. batteries

4. cosmetics

5. meds/vitamins binder

6. carpets, fabrics

7. glutathione

8. crayons

9. Paint/glue

How does corn influence you? Consider quick food for dinner. You order a cheeseburger, fries, and big Coke at the counter (or drive-through in the suburbs). You tell yourself, "No corn." All that contains corn. Deconstruct:

Cows fed corn produce meat and cheese. Meat and cheese were bonded with corn syrup and starch (same). The bun (corn flour and dextrose) and fries were fried in maize oil. High fructose corn syrup sweetens the drink and helps make the cup and straw.

Just about everything contains corn. Then what? A cornspiracy, perhaps? Is eating too much maize an issue, or should we strive to stay away from it whenever possible?

As I've said, eating some maize can be healthy. 92% of U.S. corn is genetically modified, according to the Center for Food Safety. The adjustments are expected to boost corn yields. Some sweet corn is genetically modified to produce its own insecticide, a protein deadly to insects made by Bacillus thuringiensis. It's safe to eat in sweet corn. Concerns exist about feeding agricultural animals so much maize, modified or not.

High fructose corn syrup should be consumed in moderation. Fructose, a sugar, isn't easily metabolized. Fructose causes diabetes, fatty liver, obesity, and heart disease. It causes inflammation, which might aggravate gout. Candy, packaged sweets, soda, fast food, juice drinks, ice cream, ice cream topping syrups, sauces & condiments, jams, bread, crackers, and pancake syrup contain the most high fructose corn syrup. Everyday foods with little nutrients. Check labels and choose cane sugar or sucrose-sweetened goods. Or, eat corn like the Corn Kid.

UST depegs and LUNA crashes 99.999% in a fraction of the time it takes the Moon to orbit the Earth.

Fat Man, a Terra whistle-blower, promises to expose Do Kwon's dirty secrets and shady deals.

The Terra community has voted to relaunch Terra LUNA on a new blockchain. The Terra 2.0 Pheonix-1 blockchain went live on May 28, 2022, and people were airdropped the new LUNA, now called LUNA, while the old LUNA became LUNA Classic.

Does LUNA deserve another chance? To answer this, or at least start a conversation about the Terra 2.0 chain's advantages and limitations, we must assess its fundamentals, ideology, and long-term vision.

Whatever the result, our analysis must be thorough and ruthless. A failure of this magnitude cannot happen again, so we must magnify every potential breaking point by 10.

Will UST and LUNA holders be compensated in full?

The obvious. First, and arguably most important, is to restore previous UST and LUNA holders' bags.

Terra 2.0 has 1,000,000,000,000 tokens to distribute.

• 25% of a community pool

• Holders of pre-attack LUNA: 35%

• 10% of aUST holders prior to attack

• Holders of LUNA after an attack: 10%

• UST holders as of the attack: 20%

Every LUNA and UST holder has been compensated according to the above proposal.

According to self-reported data, the new chain has 210.000.000 tokens and a $1.3bn marketcap. LUNC and UST alone lost $40bn. The new token must fill this gap. Since launch:

LUNA holders collectively own $1b worth of LUNA if we subtract the 25% community pool airdrop from the current market cap and assume airdropped LUNA was never sold.

At the current supply, the chain must grow 40 times to compensate holders. At the current supply, LUNA must reach $240.

LUNA needs a full-on Bull Market to make LUNC and UST holders whole.

Who knows if you'll be whole? From the time you bought to the amount and price, there are too many variables to determine if Terra can cover individual losses.

The above distribution doesn't consider individual cases. Terra didn't solve individual cases. It would have been huge.

What does LUNA offer in terms of value?

UST's marketcap peaked at $18bn, while LUNC's was $41bn. LUNC and UST drove the Terra chain's value.

After it was confirmed (again) that algorithmic stablecoins are bad, Terra 2.0 will no longer support them.

Algorithmic stablecoins contributed greatly to Terra's growth and value proposition. Terra 2.0 has no product without algorithmic stablecoins.

Terra 2.0 has an identity crisis because it has no actual product. It's like Volkswagen faking carbon emission results and then stopping car production.

A project that has already lost the trust of its users and nearly all of its value cannot survive without a clear and in-demand use case.

Do Kwon, how about him?

Oh, the Twitter-caller-poor? Who challenges crypto billionaires to break his LUNA chain? Who dissolved Terra Labs South Korea before depeg? Arrogant guy?

That's not a good image for LUNA, especially when making amends. I think he should step down and let a nicer person be Terra 2.0's frontman.

The verdict

Terra has a terrific community with an arrogant, unlikeable leader. The new LUNA chain must grow 40 times before it can start making up its losses, and even then, not everyone's losses will be covered.

I won't invest in Terra 2.0 or other algorithmic stablecoins in the near future. I won't be near any Do Kwon-related project within 100 miles. My opinion.

Can Terra 2.0 be saved? Comment below.