You'll start by creating an Airtable base. This will be the place where you'll send your scraped data.\
\
Navigate to [AIrtable](https://airtable.com/)

From the Airtable dashboard, click '**+Create**' from the left-hand menu.

Start from scratch

Build your ideal workflow starting with a blank table.

Change your Airtable base name to '**Top Stories**'.

Sign in to your [Browse AI](https://www.browse.ai/?utm_source=sponsored&utm_medium=email&utm_campaign=100daysofai) dashboard and click '**+ Build New Robot**'.

Enter the URL of the URL you'd like to scrape and then click "**Start Training Robot**"\
\
You're welcome to use any URL you like, but keep it simple as you're learning and try to select a website with a simple list of items.\
\
If you want to follow along with this specific example, enter [https://www.theverge.com/](https://www.theverge.com)

A new browser window will open with your URL and Browse AI's robot will appear in the top right hand side of this window.\
\
Click the robot and select the '**Capture List**' option.

Hover over the items on the page until they appear as listed items (as shown below).\
\
This is selecting the area that we want the Browse AI robot to scrape data from.

Now hover over the title until you see the option to select '**Visible Text, Link**', these are the two elements we want to scrape.\
\
This can be slightly fiddly because there are lots of elements on the page that Browse AI is able to capture, so take your time to make sure you've selected these items.

You'll now be prompted to select the first element you want to scrape.\
\
Select the '**Capture visible text**' item.

Next, click the '**Capture link**' item.\
\
Press '**Enter**' on your keyboard as this will finish the process of selecting the items you want to scrape.

You'll now be prompted to give the items you've selected a name.\
\
Give your first item the name '**Story Title**' and click the ✅ when it appears.

Repeat this process for your second item and call it '**Story Link**'.

A screen will now appear showing the structure of the data that you've selected to capture. If everything looks as expected, start by naming this list of data.\
\
Call it 'The Verge Top Stories'.

Set the number of items you want to extract as '10' and from the drop-down menu select 'There are no more items to load. \
\
Click '**Capture List**'.

You'll now be taken back to the previous screen where you can finish the recording.\
\
Click '**Finish Recording**'.

You'll be taken back to the Browse AI dashboard where you need to name your robot.\
\
Replace the text in the '**Robot name**' field. If you're following along with this example, call it 'Top Stories from The Verge'.

You'll now see that the robot is working in the background, attempting to repeat the steps you've just outlined from start to finish.\
\
Leave this to run. You'll see updates are provided on the status of the task.

Review the result

The robot is currently running a task on a cloud server. Please wait.

Created by Scott Colenutt. Running the task...
a few seconds ago
theverge.com/
No credits used yet

Your task is in the queue...

Spinning up a new server on the cloud...

Opening Chrome…

Navigating to https://www.theverge.com/…

Simulating user actions...

Saving results…

The task is taking longer than usual, but that could be normal.
Let’s give it a couple of minutes. (106 seconds left)

The task is taking quite longer than usual. It is possible for a task to take this long, but it doesn’t happen often. If it doesn’t finish soon, it will time out after 15-20 minutes.

We’re here for you if you need help.

Contact Support

Once the task is complete you'll see a preview of the URL you selected and then the steps detailed on the right-hand side.\
\
Browse AI will alert you to any potential issues it's facing in following the scraping process that you've outlined.\
\
In this example for The Verge, a Cookie Consent notice is appearing for our robot that's preventing it from being able to run. Click '**Skip and Retry**' (if it appears), as this will instruct our robot to skip any interaction with the Cookie Consent pop-up and go straight through the task steps.

The process will now re-run and you'll know when the task has been completed without issue because a '**Finished successfully**' message will appear.

Scroll to the bottom of the page and **click 'Yes, looks good'** if everything is looking as expected.

You'll now be taken to a screen where you're prompted to create optional-follow ups for your process. \
\
Click '**Add Integrations**'.\
\
This is where you'll connect Airtable.

Connect to Google Sheets, Zapier, or other API's to store or use your data

Now you're going to set your robot to scrape data and send it to Airtable at the same time, every day.\
\
Click "**Monitor**"

Uncheck this box because you're going to set the scraper to collect data at the same time every day.

In this example, set the time to 23.59, so you can collect the top 5 stories from today.\
\
The monitoring settings are already configured to run once a day, every day.

At around
Anytime
Africa/Abidjan
Africa/Accra
Africa/Addis Ababa
Africa/Algiers
Africa/Asmara
Africa/Asmera
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau
Africa/Blantyre
Africa/Brazzaville
Africa/Bujumbura
Africa/Cairo
Africa/Casablanca
Africa/Ceuta
Africa/Conakry
Africa/Dakar
Africa/Dar es Salaam
Africa/Djibouti
Africa/Douala
Africa/El Aaiun
Africa/Freetown
Africa/Gaborone
Africa/Harare
Africa/Johannesburg
Africa/Juba
Africa/Kampala
Africa/Khartoum
Africa/Kigali
Africa/Kinshasa
Africa/Lagos
Africa/Libreville
Africa/Lome
Africa/Luanda
Africa/Lubumbashi
Africa/Lusaka
Africa/Malabo
Africa/Maputo
Africa/Maseru
Africa/Mbabane
Africa/Mogadishu
Africa/Monrovia
Africa/Nairobi
Africa/Ndjamena
Africa/Niamey
Africa/Nouakchott
Africa/Ouagadougou
Africa/Porto-Novo
Africa/Sao Tome
Africa/Timbuktu
Africa/Tripoli
Africa/Tunis
Africa/Windhoek
America/Adak
America/Anchorage
America/Anguilla
America/Antigua
America/Araguaina
America/Argentina/Buenos Aires
America/Argentina/Catamarca
America/Argentina/ComodRivadavia
America/Argentina/Cordoba
America/Argentina/Jujuy
America/Argentina/La Rioja
America/Argentina/Mendoza
America/Argentina/Rio Gallegos
America/Argentina/Salta
America/Argentina/San Juan
America/Argentina/San Luis
America/Argentina/Tucuman
America/Argentina/Ushuaia
America/Aruba
America/Asuncion
America/Atikokan
America/Atka
America/Bahia
America/Bahia Banderas
America/Barbados
America/Belem
America/Belize
America/Blanc-Sablon
America/Boa Vista
America/Bogota
America/Boise
America/Buenos Aires
America/Cambridge Bay
America/Campo Grande
America/Cancun
America/Caracas
America/Catamarca
America/Cayenne
America/Cayman
America/Chicago
America/Chihuahua
America/Ciudad Juarez
America/Coral Harbour
America/Cordoba
America/Costa Rica
America/Creston
America/Cuiaba
America/Curacao
America/Danmarkshavn
America/Dawson
America/Dawson Creek
America/Denver
America/Detroit
America/Dominica
America/Edmonton
America/Eirunepe
America/El Salvador
America/Ensenada
America/Fort Nelson
America/Fort Wayne
America/Fortaleza
America/Glac

Click the "**The Verge Top Stories Limit**" field and change the limit from 10 to 5. \
\
This is just because in this example, you only want to collect the top 5 stories each day.

Double-click the "**Monitor name**" field and call this 'Top 5 Daily Stories'.

Uncheck the '**Notify me by email...**' box.\
\
In this example, the information is going to be sent straight to Airtable, so you won't need to be notified of the information via email.

Run once every
Minute
Hour
Day
Week
Month
On
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday
Deselect All
At around
Anytime
Africa/Abidjan
Africa/Accra
Africa/Addis Ababa
Africa/Algiers
Africa/Asmara
Africa/Asmera
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau
Africa/Blantyre
Africa/Brazzaville
Africa/Bujumbura
Africa/Cairo
Africa/Casablanca
Africa/Ceuta
Africa/Conakry
Africa/Dakar
Africa/Dar es Salaam
Africa/Djibouti
Africa/Douala
Africa/El Aaiun
Africa/Freetown
Africa/Gaborone
Africa/Harare
Africa/Johannesburg
Africa/Juba
Africa/Kampala
Africa/Khartoum
Africa/Kigali
Africa/Kinshasa
Africa/Lagos
Africa/Libreville
Africa/Lome
Africa/Luanda
Africa/Lubumbashi
Africa/Lusaka
Africa/Malabo
Africa/Maputo
Africa/Maseru
Africa/Mbabane
Africa/Mogadishu
Africa/Monrovia
Africa/Nairobi
Africa/Ndjamena
Africa/Niamey
Africa/Nouakchott
Africa/Ouagadougou
Africa/Porto-Novo
Africa/Sao Tome
Africa/Timbuktu
Africa/Tripoli
Africa/Tunis
Africa/Windhoek
America/Adak
America/Anchorage
America/Anguilla
America/Antigua
America/Araguaina
America/Argentina/Buenos Aires
America/Argentina/Catamarca
America/Argentina/ComodRivadavia
America/Argentina/Cordoba
America/Argentina/Jujuy
America/Argentina/La Rioja
America/Argentina/Mendoza
America/Argentina/Rio Gallegos
America/Argentina/Salta
America/Argentina/San Juan
America/Argentina/San Luis
America/Argentina/Tucuman
America/Argentina/Ushuaia
America/Aruba
America/Asuncion
America/Atikokan
America/Atka
America/Bahia
America/Bahia Banderas
America/Barbados
America/Belem
America/Belize
America/Blanc-Sablon
America/Boa Vista
America/Bogota
America/Boise
America/Buenos Aires
America/Cambridge Bay
America/Campo Grande
America/Cancun
America/Caracas
America/Catamarca
America/Cayenne
America/Cayman
America/Chicago
America/Chihuahua
America/Ciudad Juarez
America/Coral Harbour
America/Cordoba
America/Costa Rica
America/Creston
America/Cuiaba
America/Curacao
America/Danmarkshavn
America/Dawson
America/Dawson Creek
America/Denver
America/Detroit
America/Dominica
America/Edmonton
America/

Your automation is set.\
\
Now you're just going to test everything is working as expected.\
\
Click "**Run Task**".

Change the top stories limit to 5 for your test run.

Click "**Run a Task**" and your robot will now run through the entire process.

Head back to your Airtable based and you'll see a new table has been created called "The Verge Top Stories". \
\
Click to open this table.

You'll now see your scraped data has been entered. You can see the data the information was scraped, it's position on the list of stories, the story title and the story URL.\
\
Each day the automation runs in Browse AI, you'll see this information is sent to Airtable!\
\
Now imagine this automation at scale. Perhaps you could automatically build a list of the top 5 stories from all the major tech outlets and put them into a neatly curated newsletter each day? The opportunities here with scraping are endless. \
\
Have fun!

https://www.theverge.com/
5
1
TikTok ‘ban’ passes in the House again, moving to the Senate in foreign aid package
https://www.theverge.com/2024/4/20/24135163/tiktok-ban-divest-bytedance-house-foreign-aid-package
https://dashboard.browse.ai/teams/personal/robots/4e63fe3f-cf07-4a49-a758-836e4afc2202/history/0f2cdb60-8262-48b0-993f-b6e28e69596c
https://www.theverge.com/
5
2
The little smart home platform that could
https://www.theverge.com/24135207/home-assistant-announces-open-home-foundation
https://dashboard.browse.ai/teams/personal/robots/4e63fe3f-cf07-4a49-a758-836e4afc2202/history/0f2cdb60-8262-48b0-993f-b6e28e69596c
https://www.theverge.com/
5
3
How Phish turned Las Vegas’ Sphere into the ultimate music visualizer
https://www.theverge.com/24134861/phish-sphere-las-vegas-moment-factory-interview
https://dashboard.browse.ai/teams/personal/robots/4e63fe3f-cf07-4a49-a758-836e4afc2202/history/0f2cdb60-8262-48b0-993f-b6e28e69596c
https://www.theverge.com/
5
4
Is Crossrope’s smart jump rope worth $200?
https://www.theverge.com/24134172/crossrope-amp-jump-rope-set-review
https://dashboard.browse.ai/teams/personal/robots/4e63fe3f-cf07-4a49-a758-836e4afc2202/history/0f2cdb60-8262-48b0-993f-b6e28e69596c
https://www.theverge.com/
5
5
AirPlay turns the Delta emulator into a full-on retro console
https://www.theverge.com/2024/4/19/24134871/delta-emulator-apple-tv-airplay
https://dashboard.browse.ai/teams/personal/robots/4e63fe3f-cf07-4a49-a758-836e4afc2202/history/0f2cdb60-8262-48b0-993f-b6e28e69596c

Create an Automated Scraper