I sat down with Neil on September 23rd, 2019. The first day of fall had signaled its arrival with the first significant drop in temperature since I’d joined the company; instead of the brutal 80 to 90 degree weather I’d become used to, it had been well under 60 when I left the house, and I’d almost wished I’d brought a jacket.
It was on a much colder day – sometime in December, 2017 – that Neil tells me he first had the idea for Scraping Robot. It wasn’t a particularly complex idea, which fits what the company has become: a simple, straightforward data scraping solution that anyone can afford to take advantage of.
That’s what I’m here for, on this first day of fall, wearing a short-sleeved shirt covered in palm trees that’s clearly wildly inappropriate for the season. I’m here to help Neil Emeigh, president and CEO of Scraping Robot, tell our company’s story. It’s a fairly simple story, as these things go, but we want the world to know where we come from, to know why we do what we do, and to understand our dedication to providing the cheapest service in our industry.
As Neil describes it, there wasn’t much of a eureka moment. There was no Archimedean leap from the bathtub. It came from an observation of the kind of services offered by other data scraping companies, combined with his familiarity with the popular proxy server provider known as Blazing SEO.
“It was more like identifying an opportunity,” he says. “Scraping Robot started as a dream nearly 2 years ago in December 2017. That dream was to utilize the cost-effective resources of Blazing SEO to provide a cheaper scraping platform than any competitor on the market.”
It works like this. The web scraping industry as a whole relies on two primary operating expenses: servers and proxies. Without either of the above, scraping is capped at a speed that is well below what’s required for enterprise-level request volume.
Blazing SEO owns and operates a massive infrastructure of servers and proxies as a profitable business unto itself. When customers cancel their monthly contract with Blazing, those resources remain idle until another customer picks them up again. The company always has some unused stock that they’re not making any money off of. Neil realized it would be in Blazing’s best interest to sell these unused resources to Scraping Robot at pennies on the dollar.
Again, not an earth-shaking realization. But once the deal was negotiated and the price calculated, the results were still exciting: a data scraping service that could make a profit by only charging 1/18th of a cent per scrape.
An Ambitious Beginning
The team’s smaller than you might think: Neil himself, a single developer who has remained dedicated to the project since day one, and a few other engineers who’ve rotated into and out of the project over two years of development. 9 months ago the team also added a pair of system administrators, and as the product approached its launch date the team brought on a digital marketing specialist to write blog content and help push the company’s message. Pfft. Doesn’t even sound like a real job if you ask me.
“We made some mistakes in the approach at first,” Neil reluctantly tells me. “Because we knew the size of the scraping industry and how important it is, we were trying to make a system that could handle – metaphorically – a million requests per second.”
Although the team made significant progress with this ambitious approach, it became clear that perfecting a product of infinite size would take much longer than expected. About one year ago, the team re-evaluated based on a simple concept: systemization. The goal was no longer to build a Swiss army knife that could magically do everything from the get-go, but rather a LEGO set that could be quickly and easily built upon.
Scraping Robot is launching with the four most common web scraping tools: an Instagram scraper, a Google scraper, an HTML scraper, and an Amazon price scraper. But off of these four modules, new features can be built. Neil’s confident that the mix of dedicated and contracted developers we’ve assembled can build any new scraper in two weeks, tops, based on the existing infrastructure.
We haven’t yet built everything. But we can build anything.
The First (of Many) Successes
(Note: due to the high value we place on our customers’ privacy, we have made the decision to not share any identifiable details about the individuals we describe in the following sections, including their names or the names of the businesses they represent.)
About 10 months into development, the company gets its first beta customer. This individual – who’s still with us a full year later – is scraping public Instagram profile pages in order to perform data analytics. By identifying data about certain subsets of customers – data that comes from the captions people write and publish on public photos – this customer can generate lists of leads for businesses that target any niche, no matter how particular. They’ve even built a software tool of their own using the information we provide – information that comes from scraping no fewer than six million profiles every day.
This first, savvy customer knew that no one else would be able to handle a scraping volume that large at a price as low as ours. And the same Instagram scraping module he uses is now available to all of our customers at that same low price.
It didn’t take too long for more beta customers to follow. It’s another chilly day in February when Neil gets a message from a farm auction company which leverages Scraping Robot to scrape historical auction listings on multiple sites to find the mean, median, average, and maximum pricing for any piece of particular farm equipment. This is SR’s first “custom module” – a granular solution built for a specific customer by our team of expert devs.
Before hiring Scraping Robot, this auction company – which receives hundreds of appraisal requests per month – was manually aggregating all of their data. Now, they have said data at their fingertips in only a matter of minutes!
The next month, in March of this year, we heard from another dreamer. If you’ve ever worked in real estate, you know that sites like LoopNet do not proactively find property listings to put on their site, but instead rely on property managers to manually post said listings. One of our beta customers operates a new real estate search engine company that scrapes local listings in their state and provides real-time price and availability data to their customers.
Finally, we work with multiple Google scraping companies that use Scraping Robot to update their SERP tracking data in real time. Many companies use this data for their internal needs, others provide the information to their clients. Either way, that these customers have found out is that by leveraging Scraping Robot to do this work for them at low cost, they have saved tremendous amounts of money on proxies, servers, and developer resources.
Looking to the Future
By the time you’re reading this, Scraping Robot will have already launched with our four most popular modules. But if you think that’s the end of this scrappy little scraping company, you haven’t been paying attention.
Without revealing too much to the competitors who will no doubt soon be reading this: the next step is to add as many modules as possible to the library. Potentially hundreds. A roadmap has been made and developers are standing by to meet any demand.
After that: a more intense focus on custom solutions for enterprise-level scraping customers. When it comes to scraping, we can do it cheaper than any in-house solution – and that’s value we want to pass on to anyone who needs it.
Eventually, we want to move beyond the “simple scraping service” that started it all. Customers are already using data gathered with our resources in service of data analytics, but what if we could cut out the middleman and provide those analytics ourselves? Neil gets more excited as he starts describing ideas for this future, ranging from hiring expert data scientists to investing money in AI and machine learning algorithms that can extract value automatically.
That’ll be the day, I think – an actual Scraping Robot robot.
And so, Scraping Robot reaches the end of its beginning. At this point, I feel it might be best to conclude with Neil’s own words:
“I want to thank all of the developers, engineers, management, and other staff involved in the making of Scraping Robot. We have a fun and fruitful journey ahead of us.”
“And for those of you who are reading this: we welcome you with open arms to give us a try. 5000 free credits today. If we don’t have a module you need data for, contact us and we will get it implemented within 2 weeks tops.”
“The message of today is this: We dreamt of a scraping service that was simple. No confusing pricing. Modules and data that you actually need. And most importantly: the cheapest on a cost-per-scrape basis. We challenge you to compare our price with the competition: if they are saying that they can do it cheaper, get back to us and we will match it or beat it.”
“We’re that confident!”
Isaac has been writing for and about the tech industry since 2014 and has no plans to stop anytime soon. He now edits, and occasionally writes for, the Scraping Robot blog.