Join us
@dimitryzub ・ Nov 08,2021 ・ 1 min read ・ 973 views ・ Originally posted on serpapi.com
A guide to scraping the title, link, displayed link, video thumbnail, and video duration from Brave Search Organic Video results.
What is Brave Search
For the sake of non-duplicating content, I already wrote about what is Brave search in my first Brave blog post.
Intro
This blog post is a continuation of the Brave Search web scraping series. Here you’ll see how to scrape Organic Video Results from Brave Search using Python with beautifulsoup
, requests
, lxml
libraries.
Note: HTML layout might be changed in the future thus some of CSS
selectors might not work. Let me know if something isn't working.
Prerequisites
Make sure you have a basic knowledge of the libraries mentioned above, since this blog post is not exactly a tutorial for beginners, so be sure you have a basic familiarity with them. I’ll try my best to show in code that it’s not that difficult.
Also, make sure you have a basic understanding of CSS
selectors because of select()
/ select_one()
beautifulsoup
methods that accepts CSS
selectors. CSS
selectors reference.
Imports
What will be scraped
Not only 3 video results will be scraped, but 6 instead (if you click on the right arrow button), which is all in this case.
Process
Continuing to wander through Dune, let’s scrape Organic Video results about Dune.
Code is identical to scraping Brave Search News results, except we need to add video duration data and remove website source data from the output
As in the previous post, we need to find a container with needed data:
This translates to this (only id
value is changed from #news-carousel
to #video-carousel
):
After picking a container, we need to grab other elements, such as title, link, displayed link, video thumbnail and video duration with appropriate CSS
selectors.
Code
Links
Code in the online IDE • SelectorGadget
Conclusion
If you have any questions or suggestions, or something isn’t working correctly, feel free to drop a comment in the comment section.
If you want to access that feature via SerpApi, upvote on the Support Brave Search feature request, which is currently under review.
Yours,
Dimitry, and the rest of the SerpApi Team.
Join other developers and claim your FAUN account now!
Developer Advocate, SerpApi
@dimitryzubInfluence
Total Hits
Posts
Only registered users can post comments. Please, login or signup.