Buoyed by an NSF CAREER Award, Prashant Pandey looks to handle more data, more efficiently

Data is increasingly plentiful and accessible, not to mention vital to most scientific discoveries. But the traditional techniques and systems we use to manage it are often outdated or inadequate.

That’s where Prashant Pandey comes in. The Khoury assistant professor recently received the prestigious NSF CAREER Award for his work on data systems, which is helping to improve genomics, climate research, artificial intelligence, and many other fields where large amounts of data come into play.

“Are these old methods still efficient when we are dealing with humongous amounts of data?” Pandey asked. “We need to go back to the drawing board and understand the theoretical guarantees of these techniques and whether they scale at this level. If they don’t, we need to come up with new algorithmic paradigms so we can design better tools.”

One key element of Pandey’s work is ensuring that theoretical models translate into practical, scalable solutions. He explained that the goal “is to build scalable systems with strong theoretical guarantees by examining problems at the intersection of theory and practice.”

This guiding principle pervades the research associated with the NSF award. This research reimagines Bloom filters, a type of data structure that checks if an element is part of a set. These filters, Pandey explained, are “one of the most common data structures in all of computer science. They were introduced in 1970, so it’s been more than five and a half decades since they were built.”

Pandey’s research aims to make the filters more scalable, less error-prone, and better at correcting their mistakes. One key area focuses on transitioning away from traditional data analysis methods like BLAST. The tool, which stands for basic local alignment search tool, is widely used to search through genomic data.

“This is a technique that was introduced a couple of decades back and it’s very efficient,” Pandey explained. “But it can search only 1% or 2% of all the raw sequencing data that we have available, so we need to build new techniques to search the unassembled data directly and answer the big questions in biology.”

Another cornerstone of Pandey’s work is democratizing access to advanced data systems. Recently, he and his team wrote a proposal outlining a plan for a common data analysis infrastructure that would be available even to smaller labs with relatively little funding.

“There are lots of smaller labs that still want to answer those biological questions, but they cannot afford to invest money into building such heavy infrastructure,” Pandey noted. “We do not want science to suffer because of the infrastructure challenges. We want everyone to be able to answer deep biological questions and not be bottlenecked by the amount of money it takes to build this infrastructure.”

Pandey has also been researching energy efficiency in algorithms, leading to exciting advancements in how supercomputers operate.

“Given the demands of supercomputers, which are power hungry in terms of how much energy they consume to run computations, we are rethinking data structures and algorithms in terms of their energy efficiency and some of the inherent trade-offs,” Pandey said.

This means weighing the pros and cons of a task taking a few more seconds but requiring less energy, and vice versa.

The increased funding from the NSF Career Award is not the only boost for Pandey’s work. He stressed that his “students, mentors, advisors, and collaborators all make up this ecosystem around me that is enabling me to continue doing this work.”

And given the countless applications for the work, Pandey hopes that the upshot will be the advancement not just of one field or industry, but of many.

“There’s a ton of genomics data that has been produced in the last few years. The same thing applies to data generated by LLMs and new AI technologies. The same is true for other fields of science, like quantum physics or quantum chemistry,” Pandey explained. “Science should not suffer due to the lack of technical tools to handle the data. We want to enable science; that’s the high-level goal in my lab.”