AI Research Tools: Hype Or Help?

It’s time for a balanced discussion

By: Derek Jansen (MBA) | Reviewers: Kerryn Warren (PhD), David Phair (PhD) | October 2023

There’s a lot of talk about AI research tools at the moment. If the YouTubers, the bloggers and the broader mediasphere are to be believed, these AI-powered tools will (or already have) “revolutionised” research and “changed the research game”.

But how much of this is just the standard hype cycle that accompanies any new technical innovation? In this post, we’ll cut through the hype and discuss what these AI research tools can and can’t do – and how you can make the best use of the current tech.

So, where are we right now?

Unless you’ve been hiding under a rock for the last year or so, you’ll know that the sudden rise of ChatGPT has been accompanied by the birth of an entire industry of so-called “AI apps”. This explosion has received a wealth of media attention (looking at you, YouTubers…) and suddenly it seems that almost no industry is safe from the disruptive force that is AI – including academic research.

No doubt, generative AI (in other words, AI that generates content in response to a prompt) is pretty damn impressive, especially for a technology that’s still relatively new. But it’s important to recognise that many AI-based technologies have been working away under the hood for quite some time already, in many cases without us knowing about it.

The predictive text function on your smartphone (as imperfect as it may be) is a form of AI. Virtual assistants like Alexa, Siri and Cortana use AI to understand and respond to voice commands. Similarly, navigation apps like Google Maps and Waze use AI to predict traffic patterns, suggest alternate routes, and estimate travel times. So, while it may feel like the world has suddenly been flipped over, this is all part of a longer-term trend.

So, what does this mean for research? More specifically, what does this mean for you as a researcher, writing your dissertation or thesis? Well, it’s certainly not the case that you can pour a glass of your favourite red wine and kick back while your dedicated AI bot writes your dissertation – at least not yet. Even if you could, it’s pretty much a certainty that your university wouldn’t be too happy about it.

At the same time, AI-based research tools and apps do open up some interesting possibilities and AI is, in all likelihood, here to stay. As a result, it’s worth exploring how you could potentially leverage this technology to optimise your approach – ethically and sensibly. So, in this post, we’ll try to bring some balance to the current frenzy that surrounds the topic. Grab a cup of coffee (or that glass of wine) and let’s dig into it.

What AI research tools can’t do

Before looking at the opportunities that AI tools (potentially) present for researchers, we need to take a step back and understand what these tools can’t do. There are two components here – university policies and technological capabilities.

Know the rules before you play

Before you invest any amount of time running down Avenue Le AI, it’s essential to understand what your university’s policy is regarding all things AI-related. Working with our clients, we’ve observed a variety of responses from the universities. On one side of the spectrum, many institutions have outright banned the use of all AI tools, which is quite extreme considering that even something as commonplace as Grammarly would fall under this ban.

On the other end of the spectrum, some universities are allowing students to use AI tools to source information, provided that students cite AI-generated content and that such content is limited to a certain percentage. AI-generated content is a prickly topic because it forms part of a broader ethical discussion regarding ownership of IP. Specifically, who actually owns AI-generated content – the student, the AI or the creators of the data that the AI was trained on? And how do you go about citing those underlying sources when many tools don’t even disclose their data sources? Yeah, it’s muddy territory…

Long story short, every university is different and will develop its own policies regarding AI-based support. Some will embrace it, some won’t. What’s important is that you fully understand what the rules are in your case. If you’re unsure, ask for clarity. There are already cases of universities accusing students of academic misconduct (related to AI use) where the students involved didn’t even realise they were crossing the line. So, make sure that you have an up-to-date understanding of your institution’s position. Related to this, you’ll probably want to get a feel for where your research supervisor stands regarding AI – if they’re dead-set against it, you’re not going to make a great impression if you keep drumming on about your arsenal of AI tools.

Don’t fall for the hype

While the capabilities of AI are admittedly mind-boggling (at least in some use cases), it’s important to understand that despite what the well-polished websites might profess, the technology is far from perfect (at least currently). Therefore, you still need to check and double-check pretty much everything that the tools spit out.

Generative AI suffers from various shortcomings. One of the most common issues is that of bias resulting from the dataset on which the AI model is trained. In other words, biased data in, biased content out. The same applies to outdated training data. Simply put, the output you receive will be a product of the dataset that the AI tool was trained on, and in many cases, you have no real insight as to what that dataset is.

Ultimately, generative AI tools simply predict the next word (or sequence of words) based on the context provided by the preceding words in a given input text. Granted, there’s some amazing processing going on under the hood, but it’s important to recognise that the AI doesn’t “understand” what it’s outputting (at least not in the way a human would); it’s merely providing a prediction based on the patterns it observed in its training data. As a result, generative AI models can (and do) suffer from “hallucinations”, where the output can sound plausible, but is simply not grounded in reality. For example, ChatGPT has been known to provide references (in perfect APA format) that are completely made up. You certainly don’t want one of those in your dissertation…

Long story short, as much as AI tools have the potential to assist you with certain parts of the research process (and we’ll explore this next), pretty much every task still needs to start and end with you. You need to ask the right questions or provide the right instructions (this is called prompt engineering and is a rapidly developing area of specialisation). Moreover, you need to carefully check and double-check the outputs you receive. The tools, at least in their current form, will not coach you or tell you what you’re overlooking or misunderstanding. So, it’s best to view any given tool as a (highly fallible) Junior Assistant – not someone to entrust your entire research project to.

What AI research tools can help with

Now that we’ve laid a foundational disclaimer, let’s look at what sorts of tasks could potentially be aided by the use of AI research tools. Again, keep in mind that universities vary in terms of what they consider ethical use of AI tools. So, even if the tools we discuss here look like they could be helpful, be sure to double-check with your institution beforehand. You don’t want to end up losing your degree because you used a tool to shave an hour off of a multi-month process.

We’ll break this section into four broad categories:

1. Finding/identifying relevant literature
2. Evaluating sources
3. Building a literature catalogue and synthesising the information
4. Improving your writing

So, without further ado, let’s jump into it.

Task 1 – Identifying sources

A core task within any research project is finding relevant literature that will form the foundation of your study. In other words, the resources that will typically feature in your introduction and literature review chapter. To this end, there are a few AI research tools that can potentially help speed up this process.

Scite Assistant, which markets itself as “your AI-powered research assistant”, provides a chat-based interface similar to ChatGPT, but draws on academic literature to provide answers to your questions. So, you could, for example, ask it to give you an introductory overview of your research area and it would not only provide you with a response but also list a set of references for the various statements it makes (and they’ll be real references – take that, ChatGPT).

Consensus presents another option for finding academic sources, providing more of a search engine-type interface. Enter your research question and it will present concise summaries of various studies that it views as relevant to your specific question. The option to export the results to CSV is a handy addition, as that could feed directly into your literature catalogue.

Once you’ve collected a small base of core articles, Research Rabbit is another tool worth exploring. Specifically, Research Rabbit allows you to upload a core set of “starter” articles (for example, seminal literature) and then provides a visual map of the related papers. This includes papers that have cited your core articles, as well as papers that have cited those papers. You can also set up alerts to be notified of new citations, which is particularly useful when your research spans a long period.

If you’re interested in visualising the relationships between various papers, Connected Papers is another option worth exploring, as it specifically focuses on visualising these relationships.

In summary, these tools all provide a potential starting point for finding literature that’s relevant to your research aims and questions. The emphasis here is on “starting point”. Using these types of tools is not a substitute for undertaking a comprehensive literature search. The results that these tools provide are far from comprehensive. For example, in the tests we ran, multiple tools completely missed out on the seminal literature regarding the topic at hand, and many lacked the most recent literature.

Simply put, you still need to do the obligatory digging to make sure you’ve identified all the relevant papers, using a combination of Google Scholar and relevant academic databases. Nevertheless, these tools can help you fast-track the broader literature sourcing process and potentially identify some literature from adjacent disciplines that you may not have considered in your literature search – and that’s certainly helpful.

Task 2 – Evaluating source quality

Once you’ve identified a comprehensive base of literature, the next step is to evaluate the quality of your sources. This is critically important, as the strength of your study depends, at least in part, on the credibility of your literature sources.

Evaluating the quality and credibility of sources is a nuanced task, and we unpack all the evaluation criteria in our best-selling course, Literature Review Bootcamp. However, a good starting point is to take a look at how each article has been cited by other works. For example, how often has it been cited, and in what context?

Scite’s “Reference Check” can be a useful starting point for evaluating your sources. With this tool, you can upload any given article and it will provide a report detailing a few useful data points. Specifically, it will detail how many times the article has been cited, as well as the context of these citations – in other words, whether the citing studies supported or contrasted the claims of the original article.

Naturally, these classifications are AI-powered and as such, are far from perfect. That said, this functionality allows you to get a quick overview of how an article was received by the broader research community. The reference check tool will also raise any “editorial concerns”. Specifically, it will identify whether any sources have been retracted since publication. Naturally, you’ll want to verify any classifications or claims that Scite makes, but this is a useful starting point to identify potential red flags.

Task 3 – Building a literature catalogue

As we’ve discussed in previously, building a comprehensive literature catalogue is essential when it comes to synthesising or making sense of all the literature (remember, you can download a free copy of our literature catalogue template here). To this end, there are some AI-based research tools that can help fast-track this process.

With Elicit, you can upload your collection of journal articles and it will then extract key pieces of information and allow you to build a spreadsheet displaying this information. For example, you could have it create columns detailing the region, population, study type, etc. Of course, you’ll need to double-check that it’s captured the correct info, but this can provide a starting point and save you a fair amount of time.

Along a similar vein, Petal allows you to upload a PDF, or collection of PDFs, and then interact with them in a chat-based interface. This can be useful for finding very specific bits of information regarding an article or set of articles – but again, you’ll want to fact-check every response to make sure that it is indeed correct.

An important disclaimer here is that while tools like Elicit and Petal can help you compile your literature catalogue, they’re not a substitute for actually reading and engaging with the articles or resources you’ll be citing. You will still need to develop a deep understanding of the literature in your field in order to present a critical synthesis in your literature review. So, don’t make the mistake of over-relying on these tools. As with any AI tool, all output needs to be verified, and there’s simply no way you can do that if you’re not intimately familiar with the literature.

Task 4 – Improving your writing

The fourth way in which you could potentially use AI to improve your research project is the writing aspect itself. Now, this is a thorny one as you can very quickly get yourself in hot water if you don’t follow your university’s policies, so make sure that you fully understand what is (and isn’t allowed).

First up is Grammarly. If you’re not already familiar with Grammarly, it’s essentially a tool that helps you improve your writing by highlighting grammar, spelling, punctuation, and style-related issues in any given piece of text. At the risk of oversimplifying it, Grammarly is spell check on steroids, as it goes beyond the basics and offers suggestions for sentence structure, word choice, and tone (note that you’ll need the pro version to access the full functionality).

Now, it’s worth noting that Grammarly has two “sides”. The side we’ve just mentioned is generally accepted and even encouraged by universities. Essentially, it’s something of a imperfect alternative to professional editing and proofreading. However, Grammarly now also has what it calls “generative AI assistance” (Grammarly GO), which is similar to ChatGPT in that it can write things for you. This is typically not allowed by universities and using this functionality could result in your document getting flagged for AI plagiarism (or what’s coming to be known as AIgiarism).

Last but not least, what would an article about AI tools be without a suggestion regarding how to use ChatGPT itself for your research project? Now, there are, of course, many ways in which you can use ChatGPT, but one option is to use it as a tool for exploring better ways to articulate your points. For example, if you’re struggling to explain an idea or concept in a clear and concise fashion, you could paste what you’ve already written into ChatGPT and use a prompt along the lines of “rewrite this paragraph in a clearer, more concise fashion”. Of course, you can’t go copy-paste that content directly into your document, but you can take inspiration from it to help inform how you could better convey and express your points.

Along a similar vein, ChatGPT is great at generating analogies and examples to illustrate points or ideas. In other words, you could use a prompt along the lines of “Give me three examples of X” or “Give me an analogy to illustrate how A impacts B. Again, you should never copy-paste this verbatim, but this sort of approach can help get the creative juices flowing and serve as inspiration for your own writing.

The bottom line…

We’ve covered a lot of ground in this post – and to be fair, it’s probably been a bit of a party pooper if you originally had high hopes for AI’s role in your research project. That said, hopefully, we’ve helped you cut through the hype and get a more balanced view of what AI research tools can, can’t and shouldn’t do – at least at the time of writing this.

Here are the key takeaways:

1. Don’t over-rely on AI-based research tools – they’re far from perfect. As a rule of thumb, only use AI tool for things that you can verify (and will verify). Remember, it’s your degree on the line. You’re taking the risk, not them.
2. Understand exactly what your university’s position is regarding the use of AI-based tools and be sure to follow the rules.
3. This is a fast-moving space. The technological limitations, as well as the associated ethical quandaries we’ve discussed, will change over time. So, stay informed.