Web scraping tutorial with RSelenium

What to do when rvest is not enough

A tutorial of browser automation in R with RSelenium and Firefox and a reproducible example.
Tutorial
Webscraping
Author
Affiliation

Alfredo Hernández Sánchez

Vilnius University

Published

April 8, 2026

Introduction

A common problem in web scraping is that the page you see in your browser is not the same as the HTML your scraper first receives. Many websites use JavaScript to populate content after the initial HTML is loaded, which means that a simple function such as rvest::read_html() may return only a partial skeleton rather than the data you want. This tutorial offers a practical introduction to scraping those dynamic pages in R with an automated Firefox browser controlled through RSelenium. It covers the main concepts, ethical considerations, minimal setup, and a reproducible example.

It is important to consider that Selenium should not be the default. In most cases, simpler tools should come first. Selenium is best treated as a fallback for tasks that require a closer approximation of a real browsing session, such as clicking, scrolling, or handling rendered-state pagination. Because it is a heavier and more invasive tool, it should be used only when there is a genuine need, and never to bypass CAPTCHAs, anti-bot systems, or access controls.

Key concepts

Web scraping and HTML

Web scraping is the practice of collecting information from web pages by reading the page source, identifying the parts that contain the data you want, and extracting those pieces into a structured format such as a data frame. In the simplest case, you download the raw HTML and parse it. In harder cases, you need a live browser because the page content is assembled after load through scripts, clicks, or scrolling.

HTML is the markup language that gives a web page its structure. It defines elements such as headings, paragraphs, links, tables, and divs through tags and attributes. Once the browser reads that HTML, it represents the page as a structured document tree. In that structure, the individual pieces are called nodes. Some nodes are elements, some are text, and some are attributes or other document parts. In web scraping, we select specific elements or nodes from the HTML document to pull what we need, such as text, tables, or links. We can target these nodes using CSS selectors. They were designed for styling, but they are also one of the main ways scrapers locate content.

Listing 1: The HTML example below has three elements: a quote, an author, and a link. Here the CSS selectors would be .quote for the quote, #author for the author, and a for the link (href). The JavaScript script changes the author text after the page loads, a simple example of dynamic content. 
<div class="quote">"Hello world"</div>
<p id="author">Ada Lovelace</p>
<a href="/authors/ada-lovelace">Read more</a>

<script>
  document.getElementById("author").textContent = "Ada Lovelace (updated by JavaScript)";
</script>

Static vs dynamic pages

A static page is one where the data you want is already present in the HTML sent by the server. In that case, a tool like rvest::read_html() is often enough. A dynamic page is different. The initial HTML may be sparse, and JavaScript fills in the content later after the browser loads the page, makes additional requests, or responds to user actions. That is why a page can look complete in your browser but seem empty when you inspect it through a basic scraper.1

Why Selenium?

Selenium is a browser automation framework that uses WebDriver to control a real browser session. A browser driver is the bridge between your code and the browser itself. It receives WebDriver commands and controls the browser on your behalf. When you tell Selenium to open a page, click a button, or read an element, those instructions pass through the driver to Firefox. Selenium becomes useful when the page depends on JavaScript, when interaction is required to reveal data, or when you want to reproduce an actual browsing path rather than only parse raw HTML.

Ethical Considerations

Ethical web scraping starts with a simple rule. Do the least invasive thing that will get the job done. Before writing a scraper, check whether the site offers an API or a downloadable dataset, and read the site’s terms and robots.txt document.

Listing 2: Minimal example of a robots.txt file 
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Sitemap: https://example.org/sitemap.xml

The robots.txt standard exists to tell automated clients which parts of a site they are requested to crawl, but it is not a form of authorization in itself, so it should be treated as one part of a broader judgment rather than as a green light for unlimited collection. In general, it is important to limit request frequency and avoid aggressive loops. Scrape only what you need, especially when personal data could be involved.2

A practical warning

If your scraper sends too many requests too quickly, opens several browser sessions at once, or retries aggressively, it can resemble abusive automation even when your intention is harmless. In practice, this may lead to rate limits, IP blocks, or service degradation for others.

With RSelenium, the ethical threshold should be higher. A browser automation workflow is heavier than a simple rvest request because it mimics a real browsing session and can generate many more page loads, clicks, and background requests. It is especially important to start small, pause between requests, identify your client clearly when possible, and stop immediately if the site signals that you are no longer welcome.

Selenium should not be used to bypass CAPTCHAs, anti-bot controls, or access restrictions. Even when the target content is public, aggressive loops, parallel sessions, or rapid-fire retries can look like abusive bot traffic and may trigger rate limiting or defensive systems designed to protect sites from denial-of-service patterns, scraping abuse, and resource exhaustion.

RSelenium Minimal Example

This section shows a minimal example of an automated browser workflow with RSelenium. We load three necesary pacakges3 and explore the (FIRSA)[https://firsa.eu] homepage with an automated Firefox session (identified with a 🤖 icon). The code screencast is shown in Figure 1. The script goes over the following steps:

  • It starts a Firefox browser session through rsDriver() and stores the client connection in remDR. This is the object that sends instructions to the browser.

  • It opens a URL with remDR$navigate().

  • It moves the mouse and double-clicks. mouseMoveToLocation(x = 0, y = 200) moves the pointer relative to its current position.

  • It scrolls down and back up using JavaScript. The commands window.scrollBy(0, 800) and window.scrollBy(0, -300) tell the browser to move the visible page vertically.

  • It finds the Posts link with XPath and clicks it. This shows one way of targeting an element by its visible text rather than by its class.

  • It returns to the previous page with goBack().

  • It then finds a Bluesky link using a CSS selector and clicks it.

  • Finally, it closes the browser session with quit() and stops the local Selenium server with remote_firefox$server$stop().

# Load minimal libraries
library(RSelenium)
library(wdman)
library(netstat)

# Define the remote browser
remote_firefox <- rsDriver(
  browser = "firefox",
  chromever = NULL,
  phantomver = NULL,
  verbose = FALSE,
  port = free_port()
)
remDR <- remote_firefox$client

# Go to website
remDR$navigate("https://firsa.eu")

# Move mouse and click
remDR$mouseMoveToLocation(x = 0, y = 200)
remDR$doubleclick()

# Scroll down by 800 pixels
remDR$executeScript("window.scrollBy(0, 800);")

# Scroll up by 300 pixels
remDR$executeScript("window.scrollBy(0, -300);")

# Find a link and click (xpath)
remDR$findElement(
  using = "xpath",
  value = "//a[normalize-space()='Posts']"
)$clickElement()

# Return to prior page
remDR$goBack()

# Find a link and click (css)
remDR$findElement(
  using = "css selector",
  value = "a[href*='bsky.app']"
)$clickElement()

# End session and stop server
remDR$quit()
remote_firefox$server$stop()
Figure 1: A short screencast showing a minimal RSelenium workflow in R: navigating to a webpage, moving through the page, clicking links, and closing the browser session.
A small warning

Browser automation can be sensitive to page layout, screen size, timing, and small changes in the site structure. On a narrower window, for example, navigation links may collapse into a menu or move to a different part of the page, which can break selectors or make mouse movements behave differently.

Making a corpus (full workflow)

A common reason to scrape the web is to build a corpus: a structured collection of texts that can later be explored, cleaned, and analyzed. In this section, we use the JavaScript version of Quotes to Scrape as a small and safe example. Here we show how Selenium can open a rendered page, wait for the content to appear, extract repeated text elements and their metadata, and combine the results across multiple pages into a single data frame. In this example, we use a headless browser, so Firefox runs without opening a visible browser window. The script goes over the following steps:

  • Define a few helper functions to safely extract text and handle missing elements.
  • Wait for the JavaScript-rendered quote blocks to appear before trying to scrape them.
  • Extract one page at a time into a tidy data frame.
  • Launch a headless Firefox session with Selenium.
  • Loop over paginated pages, scrape each one, and store the results.
  • Combine all pages into a single corpus data frame.
# -------------------------
# Rselenium Headless Example
# -------------------------

# Load the minimal packages needed for this workflow
library(RSelenium)
library(wdman)
library(netstat)
library(dplyr)

# -----------------------------------
# 1. Helper functions for extraction
# -----------------------------------

# Safely extract text from a single child element. If the element does not exist, return NA.
safe_element_text <- function(node, css) {
  tryCatch(
    node$findChildElement(using = "css selector", value = css)$getElementText()[[1]],
    error = function(e) NA_character_
  )
}

# Safely extract text from multiple child elements. This is useful for tags, where a quote can have more than one tag.
safe_elements_text <- function(node, css) {
  elems <- tryCatch(
    node$findChildElements(using = "css selector", value = css),
    error = function(e) list()
  )
  
  if (length(elems) == 0) return(NA_character_)
  
  vals <- vapply(
    elems,
    function(x) x$getElementText()[[1]],
    character(1)
  )
  
  vals <- vals[nzchar(vals)]
  if (length(vals) == 0) NA_character_ else paste(vals, collapse = ", ")
}

# -----------------------------------------
# 2. Wait until the rendered content exists
# -----------------------------------------

# Because this page is rendered with JavaScript, we do not want to scrape immediately after navigation. 
wait_for_quotes <- function(remDr, timeout = 10, poll = 0.5) {
  n_tries <- ceiling(timeout / poll)
  
  for (i in seq_len(n_tries)) {
    nodes <- tryCatch(
      remDr$findElements(using = "css selector", value = ".quote"),
      error = function(e) list()
    )
    
    if (length(nodes) > 0) return(nodes)
    Sys.sleep(poll)
  }
  
  list()
}

# ---------------------------------
# 3. Extract one page of the corpus
# ---------------------------------

# This function takes the current page and turns all quote blocks into a tidy data frame.
extract_quotes_page <- function(remDr, page_number) {
  quote_nodes <- wait_for_quotes(remDr, timeout = 10, poll = 0.5)
  
  if (length(quote_nodes) == 0) {
    return(tibble())
  }
  
  bind_rows(lapply(quote_nodes, function(q) {
    tibble(
      quote  = safe_element_text(q, ".text"),
      author = safe_element_text(q, ".author"),
      tags   = safe_elements_text(q, ".tag"),
      page   = as.character(page_number)
    )
  }))
}

# ------------------------------------------
# 4. Launch Selenium and loop over the pages
# ------------------------------------------

scrape_quotes_selenium <- function(
  base_url = "https://quotes.toscrape.com/js/",
  max_pages = 10L
) {
  
  # Start a headless Firefox session
  rD <- rsDriver(
    browser = "firefox",
    port = free_port(),
    verbose = FALSE,
    check = TRUE,
    geckover = "latest",
    chromever = NULL,
    iedrver = NULL,
    phantomver = NULL,
    extraCapabilities = list(
      "moz:firefoxOptions" = list(
        args = list("-headless")
      )
    )
  )
  
  remDr <- rD$client
  remDr$open()
  
  # Clean up the browser and Selenium server when the function ends
  on.exit({
    try(remDr$quit(), silent = TRUE)
    try(rD$server$stop(), silent = TRUE)
  }, add = TRUE)
  
  # Create an empty list to store each scraped page
  results <- vector("list", max_pages)
  
  # Loop over paginated pages
  for (page_number in seq_len(max_pages)) {
    
    page_url <- if (page_number == 1L) {
      base_url
    } else {
      paste0(base_url, "page/", page_number, "/")
    }
    
    # Open the page in Firefox
    remDr$navigate(page_url)
    
    # Small pause to avoid scraping too aggressively
    Sys.sleep(runif(1, 0.8, 1.2))
    
    # Extract the rendered content from the current page
    page_df <- extract_quotes_page(remDr, page_number)
    
    # Stop if we reached a page with no quotes
    if (nrow(page_df) == 0) {
      results <- results[seq_len(page_number - 1L)]
      break
    }
    
    # Store the page results
    results[[page_number]] <- page_df
    
    # Another small pause between pages
    Sys.sleep(runif(1, 0.4, 0.8))
  }
  
  # Combine all pages into one corpus data frame
  bind_rows(results)
}

# ----------------------------
# 5. Run the scraper
# ----------------------------

quotes_selenium_df <- scrape_quotes_selenium()
saveRDS(quotes_selenium_df, "quotes_selenium_df.rds")
quote author tags page
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” Albert Einstein change, deep-thoughts, thinking, world 1
“It is our choices, Harry, that show what we truly are, far more than our abilities.” J.K. Rowling abilities, choices 1
“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.” Albert Einstein inspirational, life, live, miracle, miracles 1
“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.” Jane Austen aliteracy, books, classic, humor 1
“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.” Marilyn Monroe be-yourself, inspirational 1
“Try not to become a man of success. Rather become a man of value.” Albert Einstein adulthood, success, value 1
“It is better to be hated for what you are than to be loved for what you are not.” André Gide life, love 1
“I have not failed. I've just found 10,000 ways that won't work.” Thomas A. Edison edison, failure, inspirational, paraphrased 1
“A woman is like a tea bag; you never know how strong it is until it's in hot water.” Eleanor Roosevelt misattributed-eleanor-roosevelt 1
“A day without sunshine is like, you know, night.” Steve Martin humor, obvious, simile 1
“This life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.” Marilyn Monroe friends, heartbreak, inspirational, life, love, sisters 2
“It takes a great deal of bravery to stand up to our enemies, but just as much to stand up to our friends.” J.K. Rowling courage, friends 2
“If you can't explain it to a six year old, you don't understand it yourself.” Albert Einstein simplicity, understand 2
“You may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not perfect—you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being human and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a part of her that she knows you can break—her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when she makes you happy, let her know when she makes you mad, and miss her when she's not there.” Bob Marley love 2
“I like nonsense, it wakes up the brain cells. Fantasy is a necessary ingredient in living.” Dr. Seuss fantasy 2
“I may not have gone where I intended to go, but I think I have ended up where I needed to be.” Douglas Adams life, navigation 2
“The opposite of love is not hate, it's indifference. The opposite of art is not ugliness, it's indifference. The opposite of faith is not heresy, it's indifference. And the opposite of life is not death, it's indifference.” Elie Wiesel activism, apathy, hate, indifference, inspirational, love, opposite, philosophy 2
“It is not a lack of love, but a lack of friendship that makes unhappy marriages.” Friedrich Nietzsche friendship, lack-of-friendship, lack-of-love, love, marriage, unhappy-marriage 2
“Good friends, good books, and a sleepy conscience: this is the ideal life.” Mark Twain books, contentment, friends, friendship, life 2
“Life is what happens to us while we are making other plans.” Allen Saunders fate, life, misattributed-john-lennon, planning, plans 2
“I love you without knowing how, or when, or from where. I love you simply, without problems or pride: I love you in this way because I do not know any other way of loving but this, in which there is no I or you, so intimate that your hand upon my chest is my hand, so intimate that when I fall asleep your eyes close.” Pablo Neruda love, poetry 3
“For every minute you are angry you lose sixty seconds of happiness.” Ralph Waldo Emerson happiness 3
“If you judge people, you have no time to love them.” Mother Teresa attributed-no-source 3
“Anyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.” Garrison Keillor humor, religion 3
“Beauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye.” Jim Henson humor 3
“Today you are You, that is truer than true. There is no one alive who is Youer than You.” Dr. Seuss comedy, life, yourself 3
“If you want your children to be intelligent, read them fairy tales. If you want them to be more intelligent, read them more fairy tales.” Albert Einstein children, fairy-tales 3
“It is impossible to live without failing at something, unless you live so cautiously that you might as well not have lived at all - in which case, you fail by default.” J.K. Rowling NA 3
“Logic will get you from A to Z; imagination will get you everywhere.” Albert Einstein imagination 3
“One good thing about music, when it hits you, you feel no pain.” Bob Marley music 3
“The more that you read, the more things you will know. The more that you learn, the more places you'll go.” Dr. Seuss learning, reading, seuss 4
“Of course it is happening inside your head, Harry, but why on earth should that mean that it is not real?” J.K. Rowling dumbledore 4
“The truth is, everyone is going to hurt you. You just got to find the ones worth suffering for.” Bob Marley friendship 4
“Not all of us can do great things. But we can do small things with great love.” Mother Teresa misattributed-to-mother-teresa, paraphrased 4
“To the well-organized mind, death is but the next great adventure.” J.K. Rowling death, inspirational 4
“All you need is love. But a little chocolate now and then doesn't hurt.” Charles M. Schulz chocolate, food, humor 4
“We read to know we're not alone.” William Nicholson misattributed-to-c-s-lewis, reading 4
“Any fool can know. The point is to understand.” Albert Einstein knowledge, learning, understanding, wisdom 4
“I have always imagined that Paradise will be a kind of library.” Jorge Luis Borges books, library 4
“It is never too late to be what you might have been.” George Eliot inspirational 4
“A reader lives a thousand lives before he dies, said Jojen. The man who never reads lives only one.” George R.R. Martin read, readers, reading, reading-books 5
“You can never get a cup of tea large enough or a book long enough to suit me.” C.S. Lewis books, inspirational, reading, tea 5
“You believe lies so you eventually learn to trust no one but yourself.” Marilyn Monroe NA 5
“If you can make a woman laugh, you can make her do anything.” Marilyn Monroe girls, love 5
“Life is like riding a bicycle. To keep your balance, you must keep moving.” Albert Einstein life, simile 5
“The real lover is the man who can thrill you by kissing your forehead or smiling into your eyes or just staring into space.” Marilyn Monroe love 5
“A wise girl kisses but doesn't love, listens but doesn't believe, and leaves before she is left.” Marilyn Monroe attributed-no-source 5
“Only in the darkness can you see the stars.” Martin Luther King Jr. hope, inspirational 5
“It matters not what someone is born, but what they grow to be.” J.K. Rowling dumbledore 5
“Love does not begin and end the way we seem to think it does. Love is a battle, love is a war; love is a growing up.” James Baldwin love 5
“There is nothing I would not do for those who are really my friends. I have no notion of loving people by halves, it is not my nature.” Jane Austen friendship, love 6
“Do one thing every day that scares you.” Eleanor Roosevelt attributed, fear, inspiration 6
“I am good, but not an angel. I do sin, but I am not the devil. I am just a small girl in a big world trying to find someone to love.” Marilyn Monroe attributed-no-source 6
“If I were not a physicist, I would probably be a musician. I often think in music. I live my daydreams in music. I see my life in terms of music.” Albert Einstein music 6
“If you only read the books that everyone else is reading, you can only think what everyone else is thinking.” Haruki Murakami books, thought 6
“The difference between genius and stupidity is: genius has its limits.” Alexandre Dumas fils misattributed-to-einstein 6
“He's like a drug for you, Bella.” Stephenie Meyer drug, romance, simile 6
“There is no friend as loyal as a book.” Ernest Hemingway books, friends, novelist-quotes 6
“When one door of happiness closes, another opens; but often we look so long at the closed door that we do not see the one which has been opened for us.” Helen Keller inspirational 6
“Life isn't about finding yourself. Life is about creating yourself.” George Bernard Shaw inspirational, life, yourself 6
“That's the problem with drinking, I thought, as I poured myself a drink. If something bad happens you drink in an attempt to forget; if something good happens you drink in order to celebrate; and if nothing happens you drink to make something happen.” Charles Bukowski alcohol 7
“You don’t forget the face of the person who was your last hope.” Suzanne Collins the-hunger-games 7
“Remember, we're madly in love, so it's all right to kiss me anytime you feel like it.” Suzanne Collins humor 7
“To love at all is to be vulnerable. Love anything and your heart will be wrung and possibly broken. If you want to make sure of keeping it intact you must give it to no one, not even an animal. Wrap it carefully round with hobbies and little luxuries; avoid all entanglements. Lock it up safe in the casket or coffin of your selfishness. But in that casket, safe, dark, motionless, airless, it will change. It will not be broken; it will become unbreakable, impenetrable, irredeemable. To love is to be vulnerable.” C.S. Lewis love 7
“Not all those who wander are lost.” J.R.R. Tolkien bilbo, journey, lost, quest, travel, wander 7
“Do not pity the dead, Harry. Pity the living, and, above all those who live without love.” J.K. Rowling live-death-love 7
“There is nothing to writing. All you do is sit down at a typewriter and bleed.” Ernest Hemingway good, writing 7
“Finish each day and be done with it. You have done what you could. Some blunders and absurdities no doubt crept in; forget them as soon as you can. Tomorrow is a new day. You shall begin it serenely and with too high a spirit to be encumbered with your old nonsense.” Ralph Waldo Emerson life, regrets 7
“I have never let my schooling interfere with my education.” Mark Twain education 7
“I have heard there are troubles of more than one kind. Some come from ahead and some come from behind. But I've bought a big bat. I'm all ready you see. Now my troubles are going to have troubles with me!” Dr. Seuss troubles 7
“If I had a flower for every time I thought of you...I could walk through my garden forever.” Alfred Tennyson friendship, love 8
“Some people never go crazy. What truly horrible lives they must lead.” Charles Bukowski humor 8
“The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.” Terry Pratchett humor, open-mind, thinking 8
“Think left and think right and think low and think high. Oh, the thinks you can think up if only you try!” Dr. Seuss humor, philosophy 8
“What really knocks me out is a book that, when you're all done reading it, you wish the author that wrote it was a terrific friend of yours and you could call him up on the phone whenever you felt like it. That doesn't happen much, though.” J.D. Salinger authors, books, literature, reading, writing 8
“The reason I talk to myself is because I’m the only one whose answers I accept.” George Carlin humor, insanity, lies, lying, self-indulgence, truth 8
“You may say I'm a dreamer, but I'm not the only one. I hope someday you'll join us. And the world will live as one.” John Lennon beatles, connection, dreamers, dreaming, dreams, hope, inspirational, peace 8
“I am free of all prejudice. I hate everyone equally. ” W.C. Fields humor, sinister 8
“The question isn't who is going to let me; it's who is going to stop me.” Ayn Rand NA 8
“′Classic′ - a book which people praise and don't read.” Mark Twain books, classic, reading 8
“Anyone who has never made a mistake has never tried anything new.” Albert Einstein mistakes 9
“A lady's imagination is very rapid; it jumps from admiration to love, from love to matrimony in a moment.” Jane Austen humor, love, romantic, women 9
“Remember, if the time should come when you have to make a choice between what is right and what is easy, remember what happened to a boy who was good, and kind, and brave, because he strayed across the path of Lord Voldemort. Remember Cedric Diggory.” J.K. Rowling integrity 9
“I declare after all there is no enjoyment like reading! How much sooner one tires of any thing than of a book! -- When I have a house of my own, I shall be miserable if I have not an excellent library.” Jane Austen books, library, reading 9
“There are few people whom I really love, and still fewer of whom I think well. The more I see of the world, the more am I dissatisfied with it; and every day confirms my belief of the inconsistency of all human characters, and of the little dependence that can be placed on the appearance of merit or sense.” Jane Austen elizabeth-bennet, jane-austen 9
“Some day you will be old enough to start reading fairy tales again.” C.S. Lewis age, fairytales, growing-up 9
“We are not necessarily doubting that God will do the best for us; we are wondering how painful the best will turn out to be.” C.S. Lewis god 9
“The fear of death follows from the fear of life. A man who lives fully is prepared to die at any time.” Mark Twain death, life 9
“A lie can travel half way around the world while the truth is putting on its shoes.” Mark Twain misattributed-mark-twain, truth 9
“I believe in Christianity as I believe that the sun has risen: not only because I see it, but because by it I see everything else.” C.S. Lewis christianity, faith, religion, sun 9
“The truth." Dumbledore sighed. "It is a beautiful and terrible thing, and should therefore be treated with great caution.” J.K. Rowling truth 10
“I'm the one that's got to die when it's time for me to die, so let me live my life the way I want to.” Jimi Hendrix death, life 10
“To die will be an awfully big adventure.” J.M. Barrie adventure, love 10
“It takes courage to grow up and become who you really are.” E.E. Cummings courage 10
“But better to get hurt by the truth than comforted with a lie.” Khaled Hosseini life 10
“You never really understand a person until you consider things from his point of view... Until you climb inside of his skin and walk around in it.” Harper Lee better-life-empathy 10
“You have to write the book that wants to be written. And if the book will be too difficult for grown-ups, then you write it for children.” Madeleine L'Engle books, children, difficult, grown-ups, write, writers, writing 10
“Never tell the truth to people who are not worthy of it.” Mark Twain truth 10
“A person's a person, no matter how small.” Dr. Seuss inspirational 10
“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.” George R.R. Martin books, mind 10

Conclusion

RSelenium is useful when a website does not simply hand over the data in its initial HTML. If the content appears only after rendering, clicking, scrolling, or client-side pagination, browser automation can make the difference between an empty scrape and a usable dataset. However, it is a fallback, not a first resort. It is slower, more fragile, and more demanding than simpler tools, so it should be used only when the structure of the page genuinely requires it. Start with the lightest approach that works, move to Selenium when there is a clear reason, and keep the workflow limited and respectful of the target website’s policies.

This post was originally published as part of the FIRSA project.

Footnotes

  1. Having a dynamic page does not always mean you need Selenium, sometimes the underlying data can still be collected with a lighter live-browser approach rather than full automation. Current rvest even includes read_html_live() for some JavaScript-rendered cases.↩︎

  2. In R, the polite package is useful precisely because it operationalizes this mindset. It checks robots.txt, caches requests, and enforces delays rather than encouraging aggressive repetition.↩︎

  3. RSelenium controls the browser, wdman helps launch the Selenium service and browser driver, and netstat is used here to pick a free local port.↩︎