Simply Indexed.
Validin provides companies a better way to index and query data lakes with our proprietary technology, Serj.
96%
Reduction in space
3.5 trillion
Records served by Serj in a single database
250 ms
Time to first response for queries
0
Shards needed, or key length/record count restrictions
Our services
About us
We provide companies a better way to index and query data lakes using Validin’s proprietary technology, Serj. Validin has seen countless companies struggle with ingestion, indexing, retention, and cost for large, immutable data streams. Existing solutions are complex, expensive to maintain and use, not well-matched to give companies the insights they need.
Why Validin?
Common pain points we solve:
Limited by short data retention windows due to cost
Difficulty extracting value or insights from large datasets
High, often unexpected costs for data storage and processing
Difficulty managing the operational burden of data infrastructure
Advantages
Working with Validin
How we're different
01
Highly scalable
Validin's proprietary tools and services are designed to handle extremely high-volume datasets, potentially helping to solve scaling issues for companies with large amounts of data. For example, we can easily index data streams exceeding 100TB/day.
02
Clear pricing
Validin's pricing model is transparent and predictable at the outset, which helps reduce data storage, processing, and querying costs.
03
Very fast
Validin provides fast access to massive datasets with millisecond response times, broadly improving query performance and throughput over typical solutions.
04
More data
Validin's solutions are so cost-effective that many companies could increase retention several times and still reduce costs by using Validin.
05
Pipeline management
Validin manages the entire data processing pipeline within your chosen infrastructure, reducing operational burden and overhead and allowing companies to access data more efficiently.
06
Predictable costs
Validin doesn't use opaque, difficult-to-convert units (WPUs, DBUs, BBQs, etc.) to price, so you won't be surprised down the road.
07
Deployment options
Validin offers various deployment options, including on-premises, within a customer's existing cloud infrastructure, or within Validin's own managed infrastructure, so we can work within your unique constraints.
08
Data experts
Whether you've got the analytics covered and just need a platform that keeps pace, or you're looking for assistance turning information into insights, Validin's team of data processing experts provides guidance and support for data access strategies.
09
Experienced team
When you need feature extraction, summaries, alerts, change detection, and other valuable insights from massive streaming and batched data sources, Validin's team has the experience you can rely on.
Stages
The Process
1
Consultation
Validin discovers your data management needs and objectives, and you learn about Validin’s services and capabilities. To do this, Validin’s team will ask questions that will be used to guide the solution design.
2
Solution Design
From the consultation, Validin designs a proposal that meets your specific needs. Validin determines the appropriate deployment options (on-prem, cloud, or managed infrastructure) and defines the necessary infrastructure components to be adapted. We provide price estimates at this point.
3
Proof of Concept
Your company provides a slice of data to Validin (usually an hour or a day of data) that Validin uses to demonstrate the solution and finalize pricing.
4
Implementation
Once you approve the POC, Validin begins implementation of the agreed-upon solution. This includes achieving access for pipeline management purposes, creating deployment specs, configuring, tweaking, and bridging implementation gaps.
5
Ongoing Support
Validin monitors the deployment post-implementation to ensure the smooth and effective operation of the data management system. In addition, Validin will periodically review the data management systems to identify areas for improvement or optimization.
Our services
Our technology
Serj’s unique technology
Validin has modular, cloud-agnostic tooling for managing massive ingestion workflows. Validin’s real secret sauce, however, is in Serj - our proprietary indexing technology. Serj indexes have incredible storage characteristics that make them suitable for being queried directly from disk or streaming storage services like AWS S3. This allows for remarkably simple deployment topologies.
Validin can use Serj to create indexes into source data, allowing complete retrieval of context for indexed fields without mutating the original data. It can also reduce total storage space when full context isn’t needed - or is required for a much shorter window.
Hierarchical data format
A proprietary tree structure dramatically reduces data duplication and is optimized for disk reading, which saves space and enables querying from low-cost storage. In addition, this structure is optimized for querying and is significantly more efficient than Apache Parquet.
Streaming Queries
Serj is designed to answer quickly and stream data back in sorted order, allowing applications to begin processing without needing to cache results locally.
Simplified Setup
Validin manages your data for you, so you can get started quickly. Your team can then focus on your business rather than managing infrastructure.
As an example, Validin’s DNS history with Serj for >3 years of data (and constantly growing).
96%
Reduction in space
3.5 trillion
Records served by Serj in this database
250 ms
Time to first response for queries
0
Shards needed, or key length/record count restrictions
Our clients
Who can we help?
Validin can help anyone with massive amounts of immutable data, especially when that data is growing constantly.
Examples of the kinds of data Validin can help with:
- Event logs
- Web, server, and service history
- Network data (perimeter activity, netflow, dns)
- Access and audit ledgers
- Financial transaction and market data
- User-generated activity
- Telemetry and sensor data
- Monitoring metrics for computing systems
Beyond simply processing, indexing, and storing your data, we can also help with:
- Aggregations, summaries, and reporting based on the source data
- Machine learning feature output for training and evaluation of machine learning systems
- Creating derivative data streams as exhaust output (e.g. “first observed”-style events)
We support common data and file types:
- CSV
- JSON / JSONL
- PCAP
- Apache Avro™
- Apache Parquet
- SQL dumps
- Your internal, custom format!
If you process more than 10GB/day of data and want to have the best chance of retaining access without cost explosion, please reach out for a consultation.
Contacts
Contact Us
Let us know how to reach you and we’ll contact you shortly
Melbourne, FL, USA
Atlanta, GA, USA
Atlanta, GA, USA
lets.talk@validin.com