Use Guidance for Generative Artificial Intelligence (AI) Tools at NU

As artificial intelligence (AI) technologies have transformed research across various fields, faculty members face new challenges and opportunities. While AI holds enormous potential to accelerate research and drive innovation, its use also raises potential ethical, academic integrity and compliance concerns. These guidelines are designed to make faculty aware of these concerns and address them before using generative AI technologies in research at Northeastern. The guidelines begin with best practices and then provide specific requirements that must be followed.

Note: these guidelines do not apply to the use of AI for administrative activities related to your job function, which is covered by the Guidelines for the Administrative Use of Artificial Intelligence at Northeastern.

Background

Generative Artificial Intelligence (AI) refers to large learning models (LLMs) that can generate high-quality text, images, and other content based on the data they were trained on. Content that AI bots are trained on comes from user-submitted text. Popular AI “bots” include ChatGPT which produces text-based responses in the form of poems, essays, articles, letters and more based on a question entered by the user. It also can provide translation and copy language style and structure. LLMs are trained to predict the most relevant sequence of words in response to a prompt.

In research, generative AI is typically used for text summarization, writing source code, content, and idea generation, helping researchers automate tasks, analyze large datasets, and develop novel solutions across various disciplines.

Note: AI tools are only able to draw on the information that was input, meaning their responses reflect the biases and limitations of the material they have been trained on. It is important to understand that responses produced by generative AI tools will likely reflect consensus beliefs, including any biases and inaccuracies that inform those beliefs.

2. Guidelines for the Use of Generative AI in Research

Many researchers are already exploring the many uses of generative AI models to facilitate their research activities, including peer review, proposal generation, and reporting on research activities.

The University expects all members of the Northeastern community conducting research to:

Understand the limitations of generative AI and the dangers of relying on it as a source of information (see the section on Responsible AI below).

Follow guidelines set by the funding agency or publisher for the allowable and unallowable uses of generative AI throughout the peer review process (see the section on Using Generative AI in the Peer Review Process below).

Generative AI used in work submitted to funding agencies and journals should be properly cited (as permitted by the sponsor/publisher) and is at the discretion of the PI (see the section on Using Generative AI to Write Grants below).

Communicate with fellow lab and project team members about the permitted uses of generative AI on all projects based on the research activity and sponsor guidelines.

In addition to addressing the best practices and expectations set forth above, researchers must also follow the specific requirements identified below.

Authorship, Use and Citation of AI Tools

If a generative AI tool (i.e., ChatGPT) is used, your research should acknowledge how it was used, even if no generative AI content was incorporated in the work. This acknowledgement should include:

Which AI tool was used

Describe how the AI tool was used

Indicate the date AI tool was accessed

Authors who use AI tools in the writing of a paper (including any part thereof), the production of images or graphical elements for the paper, or in the collection and analysis of data must be appropriately attributed in the Materials and Methods section of the paper and may not be submitted as if it were the reported author’s own work. Failure to appropriately attribute the use of generative AI will be regarded as research misconduct and fall under the University’s Policy on Research Misconduct.

Per the University’s Policy on Research Misconduct: Research Misconduct has the same definition as under federal regulations: “fabrication, falsification, plagiarism in proposing, performing, or reviewing research, or in reporting research results.” 42 C.F.R. § 93.103. It does not include honest error or honest differences in interpretations or judgments of data.

Generative AI and Intellectual Property

Any information you plug into most generative AI tools (such as the public version of ChatGPT or any OpenAI model) becomes part of their general data lake. This means that your submission becomes part of the generative AI model and is available for public consumption. This will impact various types of intellectual property differently:

Patents: putting unpublished information that supports a patent you wish to file into a generative AI tool may adversely affect your patent. While a recent court case has affirmed that patents can only be granted to humans, having your unpublished results in a public domain could lead the US Patent Office to designate the appearance of those results in a public forum as prior art.

Copyright and publication: generative AI models may violate copyright by producing works substantially similar to published works, thus infringing on copyright. Additionally, there are questions about the boundaries around copyrightable works generated by generative AI models.

Each AI tool has different policies and methods to allow you to remove material you may have input. The university recommends reviewing instructions and Terms of Service prior to using any AI tool.

Note: You are responsible for any content you produce or publish that includes AI-generated material. AI-generated content can be inaccurate, misleading, or entirely fabricated (sometimes called “hallucinations”) or may contain copyrighted material. Additionally, because generative AI relies on existing data, AI-generated content will reflect any biases in the source database. Review and validate your AI-generated content using other reliable sources before publication. 

Generative AI and Privacy

As noted above, data shared with generative AI tools leaves the control of the researcher and the University and can be used not only by the operator of the tool but the public at large. This is not consistent with obligations that attach to personal information in many jurisdictions.

If you are using proprietary or human subjects data or any other personal information, you may NOT utilize a generative AI platform. Such uses could violate University contract terms (such as a confidentiality agreement or DUA), the consent form under which the information what collected, or the privacy laws of the jurisdiction in which the individual resides (including FERPA & GDPR).

As used in the prior paragraph, “personal information” means any information that can identify a unique individual, either directly or indirectly, by reference to an identifier such as a name, identification number, location data, online identifier, or other factors specific to the individual.

Generative AI and Confidential Information

Because of the resulting loss of control over data, it is important not to enter any University confidential information into an AI product.

This includes export-controlled information, US government Controlled Unclassified Information (I.e., CUI, CDI, SSI), or any other information covered by federal regulations. Information on export-controlled information and CUI can be found here. This also includes using AI tools to review grant proposals or peer-reviewed journal submissions/papers, which is covered in more detail below.

Please refer to the University Policy on Confidentiality of University Records and Information for additional guidance.

Using Generative AI to write grants

Generative AI may be used only if the PI understands the risks involved and adheres to these Guidelines. PIs are responsible for signing off on the proposal and promising to do the work stated if funded. PIs should keep track of how/when they are using generative in a proposal. The PI is responsible for every part of the proposal content and should utilize generative AI in an appropriate way for their research and discipline.

Currently the NIH does not specifically prohibit the use of generative AI to write grants, but they state that the PI understands and assumes the risks of using an AI tool to help write an application.

Using AI in the Peer Review Process

Reviewers are trusted and required to maintain confidentiality throughout the application process. Therefore, using AI to assist in peer review would involve a breach of confidentiality. In a recent NIH guide notice, NIH confirms their stance prohibiting NIH peer scientific peer reviewers from using natural language processers, LLMs, or other generative AI technologies for analyzing and formulating peer review critiques for grant applications and R&D contract proposals. They further state that sharing content or original concepts from an NIH grant application, contract proposal, or critique to online generative AI tools violates the NIH peer review confidentiality and integrity requirements.

3. Best Practices

Machine Learning in Research

Machine learning is a subset of artificial intelligence that empowers computers to learn and make predictions from data without explicit programming or human intervention. In research, it is predominantly used for data analysis, pattern recognition, and prediction in various fields, including healthcare, finance, and natural language processing, enabling insights and automation. One process commonly used to extract patterns, trends, and insights from large datasets is data mining.

When using machine learning techniques, keep the following best practices in mind: 

Ensure the data used for training and testing is high quality, clean, and representative of the research problem.

Rigorously assess the training data for biases to actively mitigate them and avoid unfair or discriminatory outcomes. This may involve data re-sampling, re-weighting, or using fairness-aware machine learning techniques to carefully select features and consider the potential impact on fairness.

Employ model-agnostic interpretability techniques and visualization tools to gain insights into how the machine learning model makes predictions.

Apply regularization techniques to prevent overfitting and use cross-validation to assess model generalization. Regularly validate the model’s performance on unseen data.

Document data sources, preprocessing steps, and algorithms are used in the data mining process to enhance transparency and reproducibility.

Ensure compliance with University, State, and Federal regulations on data management and privacy.

Currently within the library community, the current consensus is that responsible text and data mining is covered by fair use. However, that right is often negated by restrictions in the license governing a resource. The library community is starting to see similar concerns and restrictions in licensing around the use of licensed resources in AI projects, particularly in the use of the content to build Large Language Models (LLM).

Sources Referenced by Generative AI

Generative AI models, like ChatGPT, can manufacture facts and events. Such models’ capabilities make it difficult to discern if information is false as it can invent papers and articles that are believable. A recent article from Forbes discusses how the Federal Trade Commission (FTC) opened an investigation into whether ChatGPT poses risks to consumers after allegations the chatbot provided false information that could cause “reputational harm”. Currently AI models tend to “hallucinate” or provide made-up responses when it doesn’t know the answer to a question and/or reveal personal information.

Northeastern recommends factchecking information coming from any product of any generative AI model, including the citations as they may be false.

Responsible AI (Equity, Bias Mitigation & Accountability)

Responsible AI is meant to result in technology that is also equitable and accountable. The expectation is that organizational practices are carried out in accord with “professional responsibility,”, as an approach that “aims to ensure that professionals who design, develop, or deploy AI systems and applications or AI-based products or systems, recognize their unique position on to exert influence on people, society, and the future of AI”. (NIST AI Framework)

Additionally, responsible AI should be accurate and robust, as to contribute to the validity and trustworthiness of AI systems and can be in tension with one another in AI systems. The NIST AI Framework defines accuracy (per ISO/IEC TS 5723:2022) as “closeness of results of observations, computations, or estimates to the true values or the values accepted as being true.” Measures of accuracy should consider computational-centric measures (e.g., false positive and false negative rates), human-AI teaming, and demonstrate external validity (generalizable beyond the training conditions). Accuracy measurements should always be paired with clearly defined and realistic test sets – that are representative of conditions of expected use – and details about test methodology; these should be included in associated documentation. Accuracy measurements may include disaggregation of results for different data segments.”

Lastly, responsible AI should be fair and manage harmful bias and discrimination. Understanding and creating consistent standards of fairness can be challenging to define due to considerations that perceptions of fairness vary across cultures and may differ based on the AI application. Through risk management tactics, these differences can be recognized and considered.

NIST has identified three major categories of AI bias to be considered and managed that can exist in the absence of prejudice, partiality, or discriminatory intent:

Systemic bias can be present in AI datasets, the organization norms, practices, and processes across the AI lifecycle, and the broader society that uses AI systems.

Computational and statistical bias can be present in AI data sets and algorithmic processes; often stem from systematic errors due to non-representative samples.

Human-cognitive bias relates to how an individual or group perceives AI system information to decide or fill in missing information; or how humans think about purposes and functions of an AI system.

Bias exists in many different forms and there is no escaping the fact that they influence decisions in general day-to-day actions. As a result, biases can make their way into AI systems and therefore impact harm to individuals, groups, communities, and society. Remember, AI systems learn from what is input – if discriminatory thoughts, words, phrases, etc. are input into an AI system, it will be influenced by that bias.

4. Resources on Generative AI