Python Crawl Engineer

Scrapinghub

About the Job:

Scrapinghub is looking for a Crawl Consultant to join our Crawlera team.

Crawlera is a smart downloader designed specifically for web crawling and scraping. It allows crawler developers to crawl quickly and reliably by managing thousands of proxies internally. It is part of the Scrapinghub platform, the world’s most comprehensive web crawling stack which powers crawls of over 4 billion pages a month.

Scrapinghub helps companies, ranging from Fortune 500 enterprises to early stage startups, turn web content into useful data with a cloud-based web crawling platform, off-the-shelf datasets, and turn-key web scraping services.

Join us in making the world a better place for web crawler developers with our team of top talented engineers working remotely from more than 30 countries.

Job Responsibilities: 

Your key responsibility will be to help customers and internal developers use Crawlera successfully. In order to do this, you will need to:

  • Talk with customers to:

    • Quickly understand both their objectives and their code,
    • Help them fix any underlying issues in their code,
    • Tune & configure Crawlera to best suit each customers’ specific situation.
  • Take proactive measures to measurably improve customers’ ability to successfully use Crawlera without your intervention. You’ll be given a fair amount of autonomy over this, but it will likely include:

    • Improving client libraries, documentation, etc for Crawlera.
    • Providing extended functionality via other products in the Scrapinghub platform, or popular open source frameworks.
    • Liaison with other product teams within Scrapinghub to ensure Crawlera is well integrated in the ways that best serve customers.

This role is perfect for you if:

  • Have excellent communication in written English.
  • Have a strong knowledge of Python, Linux/UNIX, HTTP and Networking.
  • Are comfortable taking ownership in business critical situations; you enjoy being the “go-to” person.
  • Take a logical, measurement-backed approach to prioritizing projects, and enjoy working with others that do the same.
  • Are naturally dissatisfied with inefficiencies and having to do the same thing more than once; you see ways to fix the underlying cause rather than only reacting to what’s in front of you.

  • Have a broad general programming background; strong problem solver.

Bonus points for:

  • Familiarity with techniques and tools for crawling, extracting and processing data, asynchronous communication and distributed systems.
  • Good spoken English and ability to get on calls with particularly high value customers.
  • Previous customer facing experience
View this job on