FinTech

Custom financial index investment service

Our client provides custom investing indexes for the financial and insurance industries. We are honored to become one of the reasons for the client's success story. Geniusee helped to tailor one of the most important parts of their direct indexing service. We strengthened their team with our profound expertise in data engineering.

Services:

Industry:

FinTech

Location:

United States

Business context

Our expertise in data engineering appealed to the client. The Geniusee team tailored a custom data-collecting pipeline that helps create financial indexes empowering end customer’s investment strategy. We also ensured data quality and preparation for further analysis. We created a custom solution for custom financial index investing to convert and extract data from subscriptions provided by financial institutions.

Key challenges:

Assure incredibly fast data processing speeds
Build a secure and trustworthy data transfer process
Ensure system stability and periodical platform updates

Work approach

SCRUM

We launched a two-week sprint, presenting a demo to the client before every release. This methodology allows us to move quickly on the software development lifecycle. We assure the delivery of new increments on each sprint and use retrospective sessions to avoid repeating mistakes.
Data pipeline

In data pipelines, the flow of data from the source to the destination is defined, along with the processes and transformations performed along the way. There is a similarity between the standard pipeline and one of the most fundamental processes in computing: Input → Processing → Output.
PoC

We decided to go with proof of concept (principle) to examine the feasibility and relevance of the pipeline. This is also needed to verify whether the product vision of both the vendor and the client match. Primarily, our attention on PoC development was devoted to testing the typical use cases of data science and engineering projects.
Batch processing

We processed data in varying batches. In small batches, such as a couple of samples, it is referred to as mini batching, but it can also be days' worth of data. By contrast, stream processing involves handling data as it arrives on a sample-by-sample basis. Backlogs are not built up periodically by the system. Processes are instead carried out in real-time.
R&D

Our client needed to store data in a specific format. We created a custom data processor to provide instant modification of information after transfer. Libraries appeared not to be a good fit. It took around a minute to conduct a complete cycle of data processing from collection to index creation. It was not rational to take that time, so we came up with solutions that took only eight seconds to conduct the whole process.
Outstaffing

In this project, collaboration was built on an outstaffing basis. Geniusee provided a remote team of data scientists and engineers, QA specialists, and DevOps engineers. This team took part in the project as part of the client’s team. This was an unusual way of doing business, as data science projects are usually conducted by outsourcing teams. This type of cooperation model was empowered by well-designed cooperation between Geniusee and our client and deep trust.

Services

Our client needed specific services in the data engineering field to provide data collection and quality assurance of information from CME*, Bloomberg, Reuters, and others. We also had to ensure fast processing of up to a few seconds, as speed means billions in the trading world. As a team, we created a secured and stable service for direct indexing strategy, working with third parties and data vendors with a lot of trust from the client.

*CME Group stands for Chicago Mercantile Exchange, Chicago Board of Trade, New York Mercantile Exchange, and The Commodity Exchange.

At this stage, the Geniusee team defined the elements of a system to build — architecture, infrastructure, components, and data. All aspects of a system are based on the requirements specified by our client. We created a workflow for data as it came to us from CME Group databases, and then we stored, processed, and analyzed it. We chose Google Cloud to guarantee security because most data came through Google Pub/Sub.

Our team developed the pipeline to process data and create custom indexes for our client. The most important part here was to create a trustworthy backup to ensure a working recovery plan in case of shutdowns or system failures. We used Python, as this language fits perfectly for data engineering projects due to its simple syntax. It makes it relatively easy to develop a prototype quickly, and speed is crucial for our client.

Data quality was our focus during this stage. It was vital to check before data ingestion. We based it on robotic comparisons of information from different sources during specific periods. To process historical data, we used batch processing. Eventually, we came up with automated checks on a daily basis. The Geniusee team also created unit tests for the delivered pipeline to ensure correct work, smooth flow, and untroubled functionality.

At this stage, our team was working on the DevOps of the pipeline. We described its architecture and the process of architecture diagram creation. We decided to rely on infrastructure as a code (IaC). By using machine-readable definition files instead of hardware configuration or interactive configuration tools, infrastructure as a code enables computer data centers to be configured and managed efficiently.

Project tech stack

As our client works with traders as end customers, we had to choose only fast and secure technologies for custom financial index investing. To satisfy those criteria and ensure a high-performance rate and asset allocation, we decided to go with cloud services.

Written on:

Google services:

Infrastructure as a code:

Terraform

AWS services:

Elastic Kubernetes Service

SimpleDB

Relational Database Service

Geniusee team

As the project centered on data engineering and required a significant amount of expertise, we decided to go with only senior specialists and our Head of Data Science and Data Engineering as a lead.

Product team

Project manager

Development team

Oles

Tech lead

The client’s company consists of people you want to grow with. Many of them hold a Ph.D. in computer science. Being a part of such an outstanding team is an honor and motivates you to evolve constantly. Some of our colleagues on this project are heads of hedge funds, so we were sure the product we were building was tailored under perfect guidance and the requirements of the market, industry stakeholders, and our end customers eventually. Another part that drove and encouraged us was the tasks — optimization and development of algorithms that work in a fraction of a second.
Daria

Project manager

Working with a completely different project that profoundly focuses on data science was enjoyable for me. We tailored regular reports and deep communication with our client. This project seems highly successful to me due to a combination of two facts: our profound expertise and trust from the client's side.

Are you looking for full cycle product development?

You found what you were looking for. Just fill out the contact us form.

Features

High speed of data processing

We developed a solution that is 7.5 times more effective than existing services available on the market. This allows to create custom indexes based on individual stocks and huge funds that are suitable for asset management, traders’ financial goals, and cost effective.

System
scalability

To ensure scalability, we based our platform on cloud services, ensuring their stability with custom-tailored backups. The system has copies in two regions to support Google Maintenance and provide 24/7 availability.

Flexibility of the solution

Trading is a highly challenging and rapidly changing industry. To guarantee our system runs along with it, we ensure its flexibility based on module structure of custom indexing platform’s pipeline.

Daily-based data quality check

We ensure data quality with automated tests based on batch processing and historical data comparisons from external vendors. We compared data from different sources to ensure that the data was correct.