How I used the DocumentCloud Scraper Add-On to scrape all of the capitol breach court documents

Sanjin Ibrahimovic
16 steps
2 minutes

Documentcloud

Copy the US DOJ's tracker for Capitol Breach cases to your clipboard.<https://www.justice.gov/usao-dc/capitol-breach-cases>

Navigate to DocumentCloud and sign in.[https://www.documentcloud.org/app?q=%2B](https://www.documentcloud.org/app?q=%2Bproject%3Ajanuary-6-cases-215637%20)

Click "+ New Project" to create a new DocumentCloud project to store your newly scraped documents.

Give your project a name and click "Create".

Click the "Add-Ons ▼" drop-down menu to pull up your pinned Add-Ons.

Click "Scraper" to pull up the Scraper Add-On run menu.

Paste the URL in the Site field.

Navigate to your newly created project. In the search bar for the project is a trailing integer, which is the project ID. Copy this to your clipboard.

Paste the project ID in the Project field for the Add-On.

Set the access level of the scraped documents in the "Access Level" field. You can make them public, private, or only visible to other users in your organization.

Set the schedule for the Scraper Add-On. \ \ If you set it to run once, it will pull the first 100 documents it sees.\ \ If you set it to hourly, it will pull up to 100 new documents each time it runs. Scraper keeps track of which documents it has already seen and doesn't scrape them twice. \ \ If a page I'm scraping has thousands of documents, I like to set it to hourly for the first couple of days to get the bulk of the documents and then I[change the frequency](https://scribehow.com/shared/How_to_Reschedule_or_Disable_a_Scheduled_Add-On__GhpkciSxSdeqvkCY6UcVlQ) to daily or weekly as only few new documents trickle in.

Click "Dispatch" to run the Add-On. Scraper will run once when you hit dispatch and then will run again following the schedule you set.

The Scraper will run for the first time, scraping the first 100 documents it finds. It will continue to run hourly until I [disable it or change the frequency](https://scribehow.com/shared/How_to_Reschedule_or_Disable_a_Scheduled_Add-On__GhpkciSxSdeqvkCY6UcVlQ).

After the Scraper completes its first run, 100 documents are scraped.

Click "Dismiss" to clear the Add-On run status bar. The Scraper continues to run hourly in the background. Over the next two days all of the documents are scraped and stored in my project.

I already scraped all >3,400 documents from this site before this tutorial. They are available here: \ [https://www.documentcloud.org/app?q=%2Bproject%3Acapitol-breach-cases-215457](https://www.documentcloud.org/app?q=%2Bproject%3Acapitol-breach-cases-215457%20)