Video: 10 Minute IT Jams - Introducing Varada

Varada wants to take the headache out of big data analytics.

Ori Riship, Vice President of Products at the data acceleration company, has shared candid insights into the mounting challenges—and potential solutions—facing today's data-driven businesses. In a detailed interview for '10 Minute IT Jams', Riship explained how Varada is helping global enterprises deliver faster, more efficient analytics as their data volumes soar to previously unimaginable levels.

Varada's focus is on organisations dealing with massive amounts of information. "We work with data-driven organisations that have at least terabytes of data stored in their data lake," said Riship. "They need to do two major business activities: one is internal analytics, and the second one is external analytics."

Internal analytics, he explained, cover everything from typical business intelligence work to deep dives by data scientists—fuel for better decisions and company improvements. External analytics, by contrast, refers to insights that drive products or generate revenue from data. "These are the products every UI actually that we see is based on some data that is being analysed by an SQL query, hitting tables and then bringing the results. That's, in a lot of cases, what customers will pay for," he said.

But as data volumes swell—sometimes to over a trillion rows per table—the costs and complexity skyrocket. Riship sketched out the nature of the problem: "If until a couple of years ago organisations worked with hundreds of millions of rows, now they're working with hundreds of billions, even trillions of rows. Imagine the compute and the cost, the data engineering and the pipelines. The Total Cost of Ownership (TCO) becomes just irrational."

The complexity doesn't only run deep; it runs wide. "Data becomes more sophisticated. Data science teams add more dimensions—more columns—so now, not only did my tables grow in depth, but also in width. I still need to do my internal and external analytics, but now not over millions but billions and billions of rows," he added.

This exponential growth leads to spiralling processing costs, bottlenecks, and slow query response times that can undermine core services and products. Riship put it plainly: "The cost of CPU and compute becomes really, really expensive and makes outward-facing features not worthwhile. There is no ROI for them."

Enter Varada's solution: an autonomous, smart indexing layer installed within the customer's own virtual private cloud. "We invented a way to do autonomous and smart indexing inside the customer VPC. We are very strict on security and not moving the data," he said.

By deeply understanding SQL queries, Varada can build an "acceleration strategy" that combines indexing and caching to ensure queries return quickly without incurring huge costs. "Every sophisticated business question will be translated into an SQL query. What we do is we understand SQL and the needs of the SQL, then index what will benefit from indexing, and manage all of this to a certain budget. That way we enable query response time which is reasonable in terms of speed and cost," Riship said.

In practical terms, this can make the difference between a sluggish user interface and a snappy, business-ready dashboard—even when crunching through billions of records. Riship demonstrated this using a relatable example: "Let's say I'm a company like a ride-sharing platform and I want to compensate my top drivers. To do this, I need to analyse 27.6 billion rows. Without Varada, to make this query happen, businesses would need to model it, copy data, and pay a lot on infrastructure—something they do on an hourly basis."

When running the query without Varada's technology, the processing time was 75 seconds. With Varada engaged, the same query completed in just 6.2 seconds. "Customer satisfaction is the first benefit. When a customer clicks on a button and has to wait for 75 seconds, they will not be happy, especially if they are business users who need to do their job," he said. "With Varada, they do not have to wait."

The technical gains are significant too. "Earlier on, we scanned 9.6 billion rows. With Varada, only 2.3 million were scanned—that is three orders of magnitude less. This is where Varada can automatically save organisations millions of dollars," he said. The technology also removes a lot of manual data engineering work usually required to achieve similar results, including the need for detailed partitioning strategies and constant pipeline management.

Riship argues this shift saves considerable resources—not only in direct query costs, but in the day-to-day work of data engineering teams. "Partitioning becomes almost obsolete. You can stop worrying about the data engineering, stop worrying about the partitioning, because everything they need will be indexed automatically without them intervening," he said.

Asked how organisations can explore Varada's offerings, Riship pointed to the company's commitment to open source tools, as well as free-to-use software for the wider data community. "We are great believers in open source, so we contribute a lot to the community. If you're a Trino or Presto user, you can just download a connector and start using it for free for life up to four nodes," he explained.

He encouraged users to engage directly. "Come to our website, join our Slack, ask as many questions as you like. We'll be more than happy to answer," he added.

Riship concluded by reiterating the mission to empower growing enterprises. "All in all, we help organisations do actual analytics on their data lakes of big data in a reasonable cost, balancing between cost and performance in a very smart way," he said. "That's in a nutshell what we do."