Octopus. Webscrapper

We developed and supported a customizable engine for collecting random data from multiple unstructured data sources for the largest venture company in the UK.

Industry FinTech

Duration: 3840 hours

Team: 5

Download PDF

Technologies utilized

Python, RestApi, Tornado,
MySql, HTTP, App Engine, Deep Crawl

Team composition

1 Java Tech/Team Lead
2 Java Developers
1 NodeJS Developer
1 DevOps

Have a similar project?

Estimate

CLIENT

Octopus Ventures is one of Europe’s largest Venture Capital teams. Headquartered in London and New York, with venture partners in San Francisco, Singapore and China, they help entrepreneurs scale globally. Their investments range from £1m for seed to around £4m for series A. In recent years, it has fluctuated from £350k to £25m.

Challenge

The client needed an efficient way to obtain relevant information on how often products were added and removed along with the price changes.

We created Octopus Webscraper, a data aggregator that parses the necessary data at a certain periodicity and adds it to the database. The client would be able to use this data to perform analytics and receive the necessary statistics.

It needed to collect key data such as bank name, product name, interest rate, minimum investment, maximum investment, notice period and account type.

Features

Project Management

We managed the project from its inception. Having elicited requirements, our team arranged the delivery and ensured the client obtained the highest professional standards.

Prototyping

To assist the client in the visual representation of the product, we created prototypes to demonstrate how the information would be collected and deployed, using the Google App Engine.

Parser

Our engineers created the parser. We ensured that it had exception handling, so tracking mechanisms could be introduced when parse errors occur.

Data Aggregation

We created a unique data aggregator engine which helped to accumulate, sort, index a detailed API for a large amount of secure data. With its help, even a nontechnical specialist can integrate with the custom data source, collect or filter the much-needed data.

Business values

Information Accumulation

This solution helped the client to accumulate information about prices across over 2000 products and 20+ websites on an ongoing basis. In the longer term, it will help them to visualize trends of how costs are changing within the market.

Product Development

We were able to create a generic product that provides the ability to scale to include additional algorithms for a custom set of sites or data sources.