Global-Class Data Management Services

How We Create Data Streaming Solutions

Klarrio is a one-stop data management services provider and cloud-native integrator/prototype shop. We leverage our vast open-source expertise towards the modern enterprise. We enable our global customers to define and accelerate their Internet of Things and Analytics strategy through adoptions of new economy business models, disruptive technologies, and cloud native new value-added services and vendors.

As a leading data management services provider, we also provide guidance throughout these cloud on-boarding activities towards the EU general data protection regulations directives and the needed data governance & privacy regulations.

Build/Partner/Acquire

Use Case: IoT Business Strategies

After establishing that our customer was committed to entering the Internet of Things Market, the next challenge was to identify the how. Should an existing platform be acquired, should a strategic partnership be pursued, or should a dedicated platform under own control be established? What were the expected investments? What would the ROI look like? Which ‘go to market’ channels would they need?

In case a dedicated platform was chosen, what would the different R&D iterations look like and how would they find the necessary skilled resources on the market?

What we did

On behalf of its customer, Klarrio conducted a Build/Partner/Acquire exercise together with a complete 3-year business plan. The results were presented at board level. The final decision resulted in a build exercise. In order to phase the design and development of the platform and allow the business plan to get validated in parallel of the build period, first a prototype platform was built in an agnostic manner on the public cloud, followed by a Beta field trial period for our customer’s clientele and partners. Throughout this period the platform was hardened and prepared for a cloud migration towards the customer’s own data center.

Skills

  • Cloud Business Models.
  • Full IoT Stack.
  • AWS EC2/S3.
  • Kafka/Spark/Mesosphere.
  • Postgres & ScyllaDB.
  • MQTT, Lora, iOS, Android, and embedded.

What we can help you achieve

Having struggled with these questions ourselves, we strongly empathize with our customers. Additionally, our pragmatic approach towards these domains often simplifies these types of strategic queries. The combination of business experience and hands-on R&D experience is one of our key strengths which we make available throughout all of our project approaches.

Design & Development Data Hub

Use Case: Technology & Design

The development of a nationwide data services hub with strict delivery timelines. The purpose of the platform is to offer a large-scale ingestion engine for multiple national data streams and to make these available towards embedded containerized application partners as well as external partners. Additionally, the hub serves as a nationwide open API/data platform for the country’s citizens.

What we did

We hosted the customer on-site towards a joint development and testing project. All components were based of multiple cloud native technologies with specific customer customization requirements in mind. Additionally, all the cloud control plane functionalities for logging, monitoring, billing, and cloud elasticity were provided by Klarrio’s core R&D team. We played a supportive role in evangelizing the premise towards the customer’s business and product/service owner.

Skills

  • Cost & Billing models
  • Kafka, Spark, DC/OS
  • Integration of multi-tenancy
  • Jenkins, Jfrog, Packer, Terraform, Ansible
  • Customer resources training
  • AWS S3/EC2

What we can help you achieve

Starting from a proposed blueprint architecture created together with the customer’s lead architecture team, we built multiple iterations from POC to Alpha, Beta, and production grade solutions. From an R&D point of view, the architecture has its roots in a public cloud while the production grade platform will be hosted in one of the customer’s own managed data centers. Our earlier exposure towards hybrid cloud solutions together with large scale ingestion expertise act as a foundation of our added value.

Scalability Optimization

Use Case: Integration Services

Assisting our customer in scaling out their existing cloud platform as well as optimizing its performance towards serviceable persistent connected ‘Things.’ Implementation of economies of scale in relation to cost versus scalability as well as extending the demo platform lifecycle. The requirement was to enhance the overall performance of the existing platform with a factor of 10 inside of a very short timeframe without increasing the monthly cloud costs.

What we did

We reworked the cluster configurations together with properly logging and measuring the necessary performance metrics in order to pinpoint the needed areas in the stack that required reworks and/or rewrites. Institutionalization of dataframes as well as mending the bridge between the customer’s data engineering and data science team were key to optimize the overall data oriented architecture. We delivered a performance optimization of > 25-times the original platform capabilities in four months’ time.

Skills

  • Zookeeper, Kafka, HDFS, Storm
  • Enablement of numerous corresponding monitoring tools
  • Identification of new Devops, logging and monitoring tools
  • Audit on existing python scripts
  • AWS EC2/S3/SPOT

What we can help you achieve

Reaching economies of scale lies at the heart of Klarrio’s DNA. Insertion of numerous Open Source components is not the challenge—making them robust and scalable is. Finetuning the cluster configurations as well as properly integrating the different technologies remains a hurdle when operational expertise is still being learned. This is where Klarrio’s multiple years of experience comes into play.

Ingest & Streaming Customizations

Use Case: Custom Builds

Our customer required a more performant ingest engine for the deployment of a large multitenant service that needed to ingest and analyze data originating of online audio transaction stores like Spotify, iTunes, and YouTube. Strict deadlines towards end customer demos were set to show our progress. The customer’s architecture team had designed a one-off solution mixing multiple technologies like Spark, Scala, ElasticSearch together with other enterprise technologies.

What we did

We reviewed and addressed the intake and ingestion procedures of the customer’s platform and re-worked the existing process in order to allow record-by-record processing during the insertion. Additionally, we rewrote the code of the overall process. The language chosen during the original architecture was SCALA. The end result of this customization project resulted in a 10-times performance increase across the total ingestion process.

Skills

  • Code review
  • SCALA
  • SPARK
  • Architectural Design
  • IBM BigInsights

What we can help you achieve

Ingest engines lie at the core of Klarrio’s expertise, as is creating a bridge between the traditional enterprise technologies and the Open Source world. Before optimization gets launched, it is key to measure, log, and analyze in order to understand the project specific performance challenges.

Implementation & Migration of Spark Notebooks

Use Case: Analytics

Our customer had several SPARK notebooks running on IBM Softlayer. These notebooks were aimed at three different business problems, namely Customer Churn, Customer Loyalty Segmentation, and After Sales Revenues. Due to fluctuations in the customer’s vendor strategy, these notebooks required a migration and re-productization on the Microsoft Azure cloud platform. Additionally, the SPARK Notebook had to be rewritten towards Zeppelin SPARK notebooks.

What we did

After having received the original SPARK notebooks from our customer, we created the necessary MS Azure HDinsight clusters containing the SPARK images as well as a customer specific MS Blob Store environment. All data and ingestion processes were moved over, together with the re-coded Zeppelin SPARK notebooks. This was followed by a customer training and hand-over.

Skills

  • IBM Softlayer (VMs and Bare Metals, ObjectStore)
  • SPARK
  • Zeppelin
  • MS Azure (HDInsight and Blob Store)
  • Customer specific ETL processes
  • Customer training

What we can help you achieve

The Klarrio team has been working with all different types of SPARK notebook types since 2012. Due to Klarrio’s cloud IAAS agnostic philosophy, migration between vendors such as Softlayer, AWS, Azure and others has been a daily exercise. Inherently Klarrio has always conducted its R&D on AWS but has, at the same time, been deploying the production variant and/or staging platform on the customer’s Cloud IAAS vendor of choice.

Analysis of Personally Identifiable Information Presence

Use Case: Security & Privacy

Our customer had ongoing internal debates and concerns about the presence of PII (Personal Identifiable Information) throughout several of its critical storage and database environments. While most members recognized the fact that in all likelihood PII existed throughout their eco-system, few understood the scope of this occurrence. Given the upcoming EU General Data Protection Regulation (GDPR) and its financial ramifications, a more objective insight was requested.

What we did

Representative data samples of the customer where moved into an ObjectStore to perform anonymization checks on the data. These were followed by indexing the data and perform a count on all the fields against the list of PII data fields as defined by the EU GDPR commission. This relatively simple approach offered an objective ruling on the amount of PII data our customer was storing and debunked the belief that no potential GDPR challenges were present inside of the organization. The exercise helped in visualizing the need for a more prudent future approach.

Skills

  • GDPR
  • IBM Softlayer
  • SPARK

What we can help you achieve

Klarrio is a firm believer that data governance, security, and privacy are highly important and should be considered carefully—not dealt with as an ‘after thought.’ Due to Klarrio’s understanding towards the responsibilities & privileges of the data processing and data collecting community, we consider leverage of new technology to create insight and awareness a key project opportunity.

Streaming your project ahead with the
IONN™ toolbox from Klarrio

IONN

Klarrio IONN™ is a best-of-breed technology toolbox that has been cultivated over years of cloud-native stack and component designs. In addition to times when we build solutions from scratch, IONN™ offers a set of proven tools that allow us to optimize and accelerate the delivery of your project without sacrificing quality in any way. Klarrio continually invests in expanding the scope of what IONN™ entails to both future-proof your technology and enrich your digital toolchain.

The future depends
on what you do today

Take Action

Our customers are the most important visitors on our premises. They are not dependent on us—we are dependent on them. They are not an interruption in our work—they are the purpose of it. They are not the outsiders in our business—they are part of it. We are not doing them a favor by serving them—they are giving us the opportunity to serve them.

Contact us