• Books Get Your Hands Dirty on Clean Architecture Stratospheric
  • Contribute Become an Author Writing Guide Author Workflow Author Payment
  • Services Book me Advertise
  • Categories Spring Boot Java Node Kotlin AWS Software Craft Simplify! Meta Book Reviews

An Introduction to Annotations and Annotation Processing in Java

  • February 5, 2022

An annotation is a construct associated with Java source code elements such as classes, methods, and variables. Annotations provide information to a program at compile time or at runtime based on which the program can take further action. An annotation processor processes these annotations at compile time or runtime to provide functionality such as code generation, error checking, etc.

The java.lang package provides some core annotations and also gives us the capability to create our custom annotations that can be processed with annotation processors.

In this article, we will discuss the topic of annotations and demonstrate the power of annotation processing with a real-world example.

Example Code

Annotation basics.

An annotation is preceded by the @ symbol. Some common examples of annotations are @Override and @SuppressWarnings . These are built-in annotations provided by Java through the java.lang package. We can further extend the core functionality to provide our custom annotations.

An annotation by itself does not perform any action. It simply provides information that can be used at compile time or runtime to perform further processing.

Let’s look at the @Override annotation as an example:

We use the @Override annotation to mark a method that exists in a parent class, but that we want to override in a child class. The above program throws an error during compile time because the getname() method in ChildClass is annotated with @Override even though it doesn’t override a method from ParentClass (because there is no getname() method in ParentClass ).

By adding the @Override annotation in ChildClass , the compiler can enforce the rule that the overriding method in the child class should have the same case-sensitive name as that in the parent class, and so the program would throw an error at compile time, thereby catching an error which could have gone undetected even at runtime.

Standard Annotations

Below are some of the most common annotations available to us. These are standard annotations that Java provides as part of the java.lang package. To see their full effect it would be best to run the code snippets from the command line since most IDEs provide their custom options that alter warning levels.

@SuppressWarnings

We can use the @SuppressWarnings annotation to indicate that warnings on code compilation should be ignored. We may want to suppress warnings that clutter up the build output. @SuppressWarnings("unchecked") , for example, suppresses warnings associated with raw types.

Let’s look at an example where we might want to use @SuppressWarnings :

If we run this program from the command-line using the compiler switch -Xlint:unchecked to receive the full warning list, we get the following message:

The above code-block is an example of legacy Java code (before Java 5), where we could have collections in which we could accidentally store mixed types of objects. To introduce compile time error checking generics were introduced. So to get this legacy code to compile without error we would change:

If we had a large legacy code base, we wouldn’t want to go in and make lots of code changes since it would mean a lot of QA regression testing. So we might want to add the @SuppressWarning annotation to the class so that the logs are not cluttered up with redundant warning messages. We would add the code as below:

Now if we compile the program, the console is free of warnings.

@Deprecated

We can use the @Deprecated annotation to mark that a method or type has been replaced with newer functionality.

IDEs make use of annotation processing to throw a warning at compile time, usually indicating the deprecated method with a strike-through to tell the developer that they shouldn’t use this method or type anymore.

The following class declares a deprecated method:

The attribute since in the annotation tells us in which version the element was deprecated, and forRemoval indicates if the element is going to be removed in the next version.

Now, calling the legacy method as below will trigger a compile time warning indicating that the method call needs to be replaced:

We already had a look at the @Override annotation above. We can use it to indicate that a method will be overriding the method with the same signature in a parent class. It is used to throw compile time errors in cases such as typos in letter-casing as in this code example:

We intended to override the getEmployeeStatus() method but we misspelled the method name. This can lead to serious bugs. The program above would compile and run without issue without catching that bug.

If we add the annotation @Override to the getemployeeStatus() method, we get a compile time error, which causes a compile error and forces us to correct the typo right away:

@FunctionalInterface

The @FunctionalInterface annotation is used to indicate that an interface cannot have more than one abstract method. The compiler throws an error in case there is more than one abstract method. Functional interfaces were introduced in Java 8, to implement Lambda expressions and to ensure that they didn’t make use of more than one method.

Even without the @FunctionalInterface annotation, the compiler will throw an error if you include more than one abstract method in the interface. So why do we need @FunctionalInterface if it is not mandatory?

Let us take the example of the code below:

If we add another method printString2() to the Print interface, the compiler or the IDE will throw an error and this will be obvious right away.

Now, what if the Print interface was in a separate module, and there was no @FunctionalInterface annotation? The developers of that other module could easily add another function to the interface and break your code. Further, now we have to figure out which of the two is the right function in our case. By adding the @FunctionalInterface annotation we get an immediate warning in the IDE, such as this:

So it is good practice to always include the @FunctionalInterface if the interface should be usable as a Lambda.

@SafeVarargs

The varargs functionality allows the creation of methods with variable arguments. Before Java 5, the only option to create methods with optional parameters was to create multiple methods, each with a different number of parameters. Varargs allows us to create a single method to handle optional parameters with syntax as below:

However, warnings are thrown when generics are used in the arguments. @SafeVarargs allows for suppression of these warnings:

In the above code, printString() and printStringVarargs() achieve the same result. Compiling the code, however, produces a warning for printStringSafeVarargs() since it used generics:

By adding the SafeVarargs annotation as below, we can get rid of the warning:

Custom Annotations

These are annotations that are custom-created to serve a particular purpose. We can create them ourselves. We can use custom annotations to

  • reduce repetition,
  • automate the generation of boilerplate code,
  • catch errors at compile time such as potential null pointer checks,
  • customize runtime behavior based on the presence of a custom annotation.

An example of a custom annotation would be this @Company annotation:

When creating multiple instances of the CustomAnnotatedEmployee class, all instances would contain the same company name and city , so wouldn’t need to add that information to the constructor anymore.

To create a custom annotation we need to declare it with the @interface keyword:

To specify information about the scope of the annotation and the area it targets, such as compile time or runtime, we need to add meta annotations to the custom annotation.

For example, to specify that the annotation applies to classes only, we need to add @Target(ElementType.TYPE) , which specifies that this annotation only applies to classes, and @Retention(RetentionPolicy.RUNTIME) , which specifies that this annotation must be available at runtime. We will discuss further details about meta annotations once we get this basic example running.

With the meta annotations, our annotation looks like this:

Next, we need to add the fields to the custom annotation. In this case, we need name and city . So we add it as below:

Putting it all together, we can create a CustomAnnotatedEmployee class and apply the annotation to it as below:

Now we can create a test class to read the @Company annotation at runtime:

This would give the output below:

So by introspecting the annotation at runtime we can access some common information of all employees and avoid a lot of repetition if we had to construct a lot of objects.

Meta Annotations

Meta annotations are annotations applied to other annotations that provide information about the annotation to the compiler or the runtime environment.

Meta annotations can answer the following questions about an annotation:

  • Can the annotation be inherited by child classes?
  • Does the annotation need to show up in the documentation?
  • Can the annotation be applied multiple times to the same element?
  • What specific element does the annotation apply to, such as class, method, field, etc.?
  • Is the annotation being processed at compile time or runtime?

By default, an annotation is not inherited from a parent class to a child class. Applying the @Inherited meta annotation to an annotation allows it to be inherited:

Since CustomAnnotatedEmployee has the @Company annotation and CustomAnnotatedManager inherits from it, the CustomAnnotatedManager class does not need to include it.

Now if we run the test for the Manager class, we still get access to the annotation information, even though the Manager class does not have the annotation:

@Documented

@Documented ensures that custom annotations show up in the JavaDocs.

Normally, when we run JavaDoc on the class CustomAnnotatedManager the annotation information would not show up in the documentation. But when we use the @Documented annotation, it will:

@Repeatable

@Repeatable allows multiple repeating custom annotations on a method, class, or field. To use the @Repeatable annotation we need to wrap the annotation in a container class which refers to it as an array:

We declare our main class as below:

If we run a test on it as below:

We get the following output which displays the value of multiple @RepeatableCompany annotations:

@Target specifies on which elements the annotation can be used, for example in the above example the annotation @Company was defined only for TYPE and so it could only be applied to a class.

Let’s see what happens if we apply the @Company annotation to a method:

If we applied the @Company annotation to the method getEmployeeStatus() as above, we get a compiler error stating: '@Company' not applicable to method.

The various self-explanatory target types are:

  • ElementType.ANNOTATION_TYPE
  • ElementType.CONSTRUCTOR
  • ElementType.FIELD
  • ElementType.LOCAL_VARIABLE
  • ElementType.METHOD
  • ElementType.PACKAGE
  • ElementType.PARAMETER
  • ElementType.TYPE

@Retention specifies when the annotation is discarded.

SOURCE - The annotation is used at compile time and discarded at runtime.

CLASS - The annotation is stored in the class file at compile time and discarded at run time.

RUNTIME - The annotation is retained at runtime.

If we needed an annotation to only provide error checking at compile time as @Override does, we would use SOURCE . If we need an annotation to provide functionality at runtime such as @Test in Junit we would use RUNTIME . To see a real example, create the following annotations in 3 separate files:

Now create a class that uses all 3 annotations:

To verify that only the runtime annotation is available at runtime, run a test as follows:

The output would be as follows:

So we verified that only the RUNTIME annotation gets processed at runtime.

Annotation Categories

Annotation categories distinguish annotations based on the number of parameters that we pass into them. By categorizing annotations as parameter-less, single value, or multi-value, we can more easily think and talk about annotations.

Marker Annotations

Marker annotations do not contain any members or data. We can use the isAnnotationPresent() method at runtime to determine the presence or absence of a marker annotation and make decisions based on the presence of the annotation.

For example, if our company had several clients with different data transfer mechanisms, we could annotate the class with an annotation indicating the method of data transfer as below:

The client class could use the annotation as below:

We can process the annotation as follows:

Based on whether the @CSV annotation exists or not, we can decide whether to write out the information to CSV or an Excel file. The above program would produce this output:

Single-Value Annotations

Single-value annotations contain only one member and the parameter is the value of the member. The single member has to be named value .

Let’s create a SingleValueAnnotationCompany annotation that uses only the value field for the name, as below:

Create a class that uses the annotation as below:

Run a test as below:

The single value ‘XYZ’ overrides the default annotation value and the output is as below:

Full Annotations

They consist of multiple name value pairs. For example Company(name="ABC", city="XYZ") . Considering our original Company example:

Let’s create the MultiValueAnnotatedEmployee class as below. Specify the parameters and values as below. The default values will be overwritten.

The output is as below, and has overridden the default annotation values:

Building a Real-World Annotation Processor

For our real-world annotation processor example, we are going to do a simple simulation of the annotation @Test in JUnit. By marking our functions with the @Test annotation we can determine at runtime which of the methods in a test class need to be run as tests.

We first create the annotation as a marker annotation for methods:

Next, we create a class AnnotatedMethods , to which we will apply the @Test annotations to the method test1() . This will enable the method to be executed at runtime. The method test2() does not have an annotation, and should not be executed at runtime.

Now we create the test to run the AnnotatedMethods class:

By calling getDeclaredMethods() , we’re getting the methods of our AnnotatedMethods class. Then, we’re iterating through the methods and checking each method if it is annotated with the @Test annotation. Finally, we perform a runtime invocation of the methods that were identified as being annotated with @Test .

We want to verify the test1() method will run since it is annotated with @Test , and test2() will not run since it is not annotated with @Test .

The output is:

So we verified that test2() , which did not have the @Test annotation, did not have its output printed.

We did an overview of annotations, followed by a simple real-world example of annotation processing.

We can further use the power of annotation processing to perform more complex automated tasks such as creating builder source files for a set of POJOs at compile time. A builder is a design pattern in Java that is used to provide a better alternative to constructors when there is a large number of parameters involved or there is a need for multiple constructors with optional parameters. If we had a few dozen POJOs, the code generation capabilities of the annotation processor would save us a lot of time by creating the corresponding builder files at compile time.

By fully leveraging the power of annotation processing we will be able to skip a lot of repetition and save a lot of time.

You can play around with the code examples from this article on GitHub .

Arshad Syed

an annotation method

As a software developer I am always interested in learning new technologies. I am currently working on improving my full stack skills, and also on teaching technology.

Recent Posts

Merge Sort in Kotlin

Merge Sort in Kotlin

  • Ezra Kanake
  • February 20, 2024

Sorting is a fundamental operation that plays a crucial role in various applications. Among the many sorting algorithms, merge sort stands out for its efficiency and simplicity.

Extension Functions in Kotlin

Extension Functions in Kotlin

  • February 13, 2024

One of Kotlin’s standout features is extension functions, a mechanism that empowers developers to enhance existing classes without modifying their source code.

Use Cases for Java Records

Use Cases for Java Records

Sachin Raverkar

  • February 11, 2024

Java Records introduce a simple syntax for creating data-centric classes, making our code more concise, expressive, and maintainable. In this guide, we’ll explore the key concepts and practical applications of Java Records, providing a step-by-step guide to creating records and sharing best practices for using them effectively in projects.

The Java Tutorials have been written for JDK 8. Examples and practices described in this page don't take advantage of improvements introduced in later releases and might use technology no longer available. See Java Language Changes for a summary of updated language features in Java SE 9 and subsequent releases. See JDK Release Notes for information about new features, enhancements, and removed or deprecated options for all JDK releases.

Annotations Basics

The format of an annotation.

In its simplest form, an annotation looks like the following:

The at sign character ( @ ) indicates to the compiler that what follows is an annotation. In the following example, the annotation's name is Override :

The annotation can include elements , which can be named or unnamed, and there are values for those elements:

If there is just one element named value , then the name can be omitted, as in:

If the annotation has no elements, then the parentheses can be omitted, as shown in the previous @Override example.

It is also possible to use multiple annotations on the same declaration:

If the annotations have the same type, then this is called a repeating annotation:

Repeating annotations are supported as of the Java SE 8 release. For more information, see Repeating Annotations .

The annotation type can be one of the types that are defined in the java.lang or java.lang.annotation packages of the Java SE API. In the previous examples, Override and SuppressWarnings are predefined Java annotations . It is also possible to define your own annotation type. The Author and Ebook annotations in the previous example are custom annotation types.

Where Annotations Can Be Used

Annotations can be applied to declarations: declarations of classes, fields, methods, and other program elements. When used on a declaration, each annotation often appears, by convention, on its own line.

As of the Java SE 8 release, annotations can also be applied to the use of types. Here are some examples:

  • Class instance creation expression: new @Interned MyObject();
  • Type cast: myString = (@NonNull String) str;
  • implements clause: class UnmodifiableList<T> implements @Readonly List<@Readonly T> { ... }
  • Thrown exception declaration: void monitorTemperature() throws @Critical TemperatureException { ... }

This form of annotation is called a type annotation . For more information, see Type Annotations and Pluggable Type Systems .

About Oracle | Contact Us | Legal Notices | Terms of Use | Your Privacy Rights

Copyright © 1995, 2022 Oracle and/or its affiliates. All rights reserved.

  • Watch & Listen
  • Oracle University

Annotations

Annotations have a number of uses, among them:

  • Information for the compiler — Annotations can be used by the compiler to detect errors or suppress warnings.
  • Compile-time and deployment-time processing — Software tools can process annotation information to generate code, XML files, and so forth.
  • Runtime processing — Some annotations are available to be examined at runtime.

This section explains where annotations can be used, how to apply annotations, what predefined annotation types are available in the Java Platform, Standard Edition (Java SE API), how type annotations can be used in conjunction with pluggable type systems to write code with stronger type checking, and how to implement repeating annotations.

The Format of an Annotation

In its simplest form, an annotation looks like the following:

The at sign character ( @ ) indicates to the compiler that what follows is an annotation. In the following example, the annotation's name is Override :

The annotation can include elements , which can be named or unnamed, and there are values for those elements:

If there is just one element named value , then the name can be omitted, as in:

If the annotation has no elements, then the parentheses can be omitted, as shown in the previous @Override example.

It is also possible to use multiple annotations on the same declaration:

If the annotations have the same type, then this is called a repeating annotation:

Repeating annotations are supported as of the Java SE 8 release. For more information, see the section Repeating Annotations.

The annotation type can be one of the types that are defined in the java.lang or java.lang.annotation packages of the Java SE API. In the previous examples, Override and SuppressWarnings are predefined Java annotations. It is also possible to define your own annotation type. The Author and Ebook annotations in the previous example are custom annotation types.

Where Annotations Can Be Used

Annotations can be applied to declarations: declarations of classes, fields, methods, and other program elements. When used on a declaration, each annotation often appears, by convention, on its own line.

As of the Java SE 8 release, annotations can also be applied to the use of types. Here are some examples:

Class instance creation expression:

implements clause:

Thrown exception declaration:

This form of annotation is called a type annotation .

Declaring an Annotation Type

Many annotations replace comments in code.

Suppose that a software group traditionally starts the body of every class with comments providing important information:

To add this same metadata with an annotation, you must first define the annotation type. The syntax for doing this is:

The annotation type definition looks similar to an interface definition where the keyword interface is preceded by the at sign ( @ ) ( @ = AT, as in annotation type). Annotation types are a form of interface, which will be covered in a later section. For the moment, you do not need to understand interfaces.

The body of the previous annotation definition contains annotation type element declarations, which look a lot like methods. Note that they can define optional default values.

After the annotation type is defined, you can use annotations of that type, with the values filled in, like this:

Note: To make the information in @ClassPreamble appear in Javadoc-generated documentation, you must annotate the @ClassPreamble definition with the @Documented annotation:

Predefined Annotation Types

A set of annotation types are predefined in the Java SE API. Some annotation types are used by the Java compiler, and some apply to other annotations.

Annotation Types Used by the Java Language

The predefined annotation types defined in java.lang are @Deprecated , @Override , and @SuppressWarnings .

@Deprecated

@Deprecated annotation indicates that the marked element is deprecated and should no longer be used. The compiler generates a warning whenever a program uses a method, class, or field with the @Deprecated annotation. When an element is deprecated, it should also be documented using the Javadoc @deprecated tag, as shown in the following example. The use of the at sign ( @ ) in both Javadoc comments and in annotations is not coincidental: they are related conceptually. Also, note that the Javadoc tag starts with a lowercase d and the annotation starts with an uppercase D .

Note that, as of Java SE 9, a forRemoval attribute has been added to the @Deprecated annotation. It indicates whether the annotated element is subject to removal in a future version. The default value is false .

@Override annotation informs the compiler that the element is meant to override an element declared in a superclass. Overriding methods will is discussed in the section Interfaces and Inheritance.

While it is not required to use this annotation when overriding a method, it helps to prevent errors. If a method marked with @Override fails to correctly override a method in one of its superclasses, the compiler generates an error.

@SuppressWarnings

@SuppressWarnings annotation tells the compiler to suppress specific warnings that it would otherwise generate. In the following example, a deprecated method is used, and the compiler usually generates a warning. In this case, however, the annotation causes the warning to be suppressed.

Every compiler warning belongs to a category. The Java Language Specification lists two categories: deprecation and unchecked. The unchecked warning can occur when interfacing with legacy code written before the advent of generics. To suppress multiple categories of warnings, use the following syntax:

@SafeVarargs

@SafeVarargs annotation, when applied to a method or constructor, asserts that the code does not perform potentially unsafe operations on its varargs parameter. When this annotation type is used, unchecked warnings relating to varargs usage are suppressed.

@FunctionalInterface

@FunctionalInterface annotation, introduced in Java SE 8, indicates that the type declaration is intended to be a functional interface, as defined by the Java Language Specification.

Annotations That Apply to Other Annotations

Annotations that apply to other annotations are called meta-annotations. There are several meta-annotation types defined in java.lang.annotation .

@Retention annotation specifies how the marked annotation is stored:

  • RetentionPolicy.SOURCE – The marked annotation is retained only in the source level and is ignored by the compiler.
  • RetentionPolicy.CLASS – The marked annotation is retained by the compiler at compile time, but is ignored by the Java Virtual Machine (JVM).
  • RetentionPolicy.RUNTIME – The marked annotation is retained by the JVM so it can be used by the runtime environment.

@Documented

@Documented annotation indicates that whenever the specified annotation is used those elements should be documented using the Javadoc tool. (By default, annotations are not included in Javadoc.) For more information, see the Javadoc tools page.

@Target annotation marks another annotation to restrict what kind of Java elements the annotation can be applied to. A target annotation specifies one of the following element types as its value:

  • ElementType.ANNOTATION_TYPE can be applied to an annotation type.
  • ElementType.CONSTRUCTOR can be applied to a constructor.
  • ElementType.FIELD can be applied to a field or property.
  • ElementType.LOCAL_VARIABLE can be applied to a local variable.
  • ElementType.METHOD can be applied to a method-level annotation.
  • ElementType.MODULE can be applied to a module declaration.
  • ElementType.PACKAGE can be applied to a package declaration.
  • ElementType.PARAMETER can be applied to the parameters of a method.
  • ElementType.RECORD_COMPONENT can be applied to the component of a record.
  • ElementType.TYPE can be applied to the declaration of a class, an abtract class, an interface, an annotation interface, an enumeration, or a record declaration.
  • ElementType.TYPE_PARAMETER can be applied on the parameters of a type.
  • ElementType.TYPE_USE can be applied where a type is used, for instance on the declaration of a field.

@Inherited annotation indicates that the annotation type can be inherited from the super class. (This is not true by default.) When the user queries the annotation type and the class has no annotation for this type, the class' superclass is queried for the annotation type. This annotation applies only to class declarations.

@Repeatable

@Repeatable annotation, introduced in Java SE 8, indicates that the marked annotation can be applied more than once to the same declaration or type use. For more information, see the section Repeating Annotations.

Type Annotations and Pluggable Type Systems

Before the Java SE 8 release, annotations could only be applied to declarations. As of the Java SE 8 release, annotations can also be applied to any type use. This means that annotations can be used anywhere you use a type. A few examples of where types are used are class instance creation expressions ( new ), casts, implements clauses, and throws clauses. This form of annotation is called a type annotation and several examples are provided in the section Annotations Basics.

Type annotations were created to support improved analysis of Java programs way of ensuring stronger type checking. The Java SE 8 release does not provide a type checking framework, but it allows you to write (or download) a type checking framework that is implemented as one or more pluggable modules that are used in conjunction with the Java compiler.

For example, you want to ensure that a particular variable in your program is never assigned to null; you want to avoid triggering a NullPointerException . You can write a custom plug-in to check for this. You would then modify your code to annotate that particular variable, indicating that it is never assigned to null. The variable declaration might look like this:

When you compile the code, including the NonNull module at the command line, the compiler prints a warning if it detects a potential problem, allowing you to modify the code to avoid the error. After you correct the code to remove all warnings, this particular error will not occur when the program runs.

You can use multiple type-checking modules where each module checks for a different kind of error. In this way, you can build on top of the Java type system, adding specific checks when and where you want them.

With the judicious use of type annotations and the presence of pluggable type checkers, you can write code that is stronger and less prone to error.

In many cases, you do not have to write your own type checking modules. There are third parties who have done the work for you. For example, you might want to take advantage of the Checker Framework created by the University of Washington. This framework includes a NonNull module, as well as a regular expression module, and a mutex lock module. For more information, see the Checker Framework .

Repeating Annotations

There are some situations where you want to apply the same annotation to a declaration or type use. As of the Java SE 8 release, repeating annotations enable you to do this.

For example, you are writing code to use a timer service that enables you to run a method at a given time or on a certain schedule, similar to the UNIX cron service. Now you want to set a timer to run a method, doPeriodicCleanup() , on the last day of the month and on every Friday at 11:00 p.m. To set the timer to run, create an @Schedule annotation and apply it twice to the doPeriodicCleanup() method. The first use specifies the last day of the month and the second specifies Friday at 11p.m., as shown in the following code example:

The previous example applies an annotation to a method. You can repeat an annotation anywhere that you would use a standard annotation. For example, you have a class for handling unauthorized access exceptions. You annotate the class with one @Alert annotation for managers and another for admins:

For compatibility reasons, repeating annotations are stored in a container annotation that is automatically generated by the Java compiler. In order for the compiler to do this, two declarations are required in your code.

Step 1: Declare a Repeatable Annotation Type

The annotation type must be marked with the @Repeatable meta-annotation. The following example defines a custom @Schedule repeatable annotation type:

The value of the @Repeatable meta-annotation, in parentheses, is the type of the container annotation that the Java compiler generates to store repeating annotations. In this example, the containing annotation type is @Schedules , so repeating @Schedule annotations is stored in an @Schedules annotation.

Applying the same annotation to a declaration without first declaring it to be repeatable results in a compile-time error.

Step 2: Declare the Containing Annotation Type

The containing annotation type must have a value element with an array type. The component type of the array type must be the repeatable annotation type. The declaration for the @Schedules containing annotation type is the following:

Retrieving Annotations

There are several methods available in the Reflection API that can be used to retrieve annotations. The behavior of the methods that return a single annotation, such as AnnotatedElement.getAnnotation(Class<T>) , are unchanged in that they only return a single annotation if one annotation of the requested type is present. If more than one annotation of the requested type is present, you can obtain them by first getting their container annotation. In this way, legacy code continues to work. Other methods were introduced in Java SE 8 that scan through the container annotation to return multiple annotations at once, such as AnnotatedElement.getAnnotationsByType(Class<T>) . See the AnnotatedElement class specification for information on all of the available methods.

Design Considerations

When designing an annotation type, you must consider the cardinality of annotations of that type. It is now possible to use an annotation zero times, once, or, if the annotation's type is marked as @Repeatable , more than once. It is also possible to restrict where an annotation type can be used by using the @Target meta-annotation. For example, you can create a repeatable annotation type that can only be used on methods and fields. It is important to design your annotation type carefully to ensure that the programmer using the annotation finds it to be as flexible and powerful as possible.

In this tutorial

Last update: September 14, 2021

  • Enterprise Java
  • Web-based Java
  • Data & Java
  • Project Management
  • Visual Basic
  • Ruby / Rails
  • Java Mobile
  • Architecture & Design
  • Open Source
  • Web Services

Developer.com

Developer.com content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More .

Annotation-based Java development is certainly one of the most notable recent development trends. Annotation-based development relieves Java developers from the pain of cumbersome configuration. First introduced in Java 5.0, annotations are one of the features in that JDK version that shifted the responsibility for writing boilerplate Java code from the programmer to the compiler. When the source code is free of boilerplate code, it becomes easier to maintain. The resulting code is also less likely to contain bugs.

Java annotations are one of the main ease-of-development features introduced in JDK 5. Annotations are like meta-tags that you can add to your code and apply to package declarations, type declarations, constructors, methods, fields, parameters and variables. They provide a helpful way to indicate whether your methods are dependent on other methods, whether they are incomplete, whether your classes have references to other classes, and so on.

Quoting from Oracle’s official site , “It [annotation-based development] lets us avoid writing boilerplate code under many circumstances by enabling tools to generate it from annotations in the source code. This leads to a declarative programming style where the programmer says what should be done and tools emit the code to do it.”

Simply speaking, an annotation is a mechanism for associating a meta-tag with program elements and allowing the compiler or the VM to extract program behaviors from these annotated elements and generate interdependent code when necessary.

In the first part of this three-article series, I’ll describe some basics of Java annotations, their benefits, as well as provide some example usages.

Java Annotations Basics

There are two things you need to consider with annotations. One is the “annotation” itself; another is the “annotation type.” An annotation is the meta-tag that you will use in your code to give it some life. Annotation type is used for defining an annotation. You will use it when you want to create your own custom annotation. The type is the actual construct used, and the annotation is the specific usage of that type.

An annotation type definition takes an “at” (@) sign, followed by the interface keyword plus the annotation name. On the other hand, an annotation takes the form of an “at” sign (@), followed by the annotation type. This is simplest form of annotation. Additionally, you can put data within parenthesis after the annotation name. An example of each can be seen below:

Example to Define an Annotation (Annotation Type)

Example to annotate your code (annotation), java annotation types.

There are three annotation types:

Marker: Marker type annotations have no elements, except the annotation name itself.

Rules of Thumb for Defining Java Annotation Types

Here are some rules-of-thumb when defining an annotation type:

  • Annotation declaration should start with an ‘at’ sign like @, following with an interface keyword, following with the annotation name.
  • Method declarations should not have any parameters.
  • Method declarations should not have any throws clauses.
  • array of the above types

There are two types of annotations available with JDK5:

  • Simple annotations: These are the basic types supplied with Tiger, which you can use to annotate your code only; you cannot use those to create a custom annotation type.
  • Meta annotations: These are the annotation types designed for annotating annotation-type declarations. Simply speaking, these are called the annotations-of-annotations.

Simple Java Annotations

There are only three types of simple annotations provided by JDK5. They are:

  • Suppresswarnings

It’s important to note that JDK5 (in other words, Tiger) actually does not have many built-in annotations; rather, it allows core Java the ability to support the annotation feature. The charter for JSR-175 strictly dictated it was to define a metadata facility. It was left to the programmers to write custom annotation types and to other JSRs to write a set of standard annotation types. The following sections will describe each simple annotation in more depth, along with examples.

The Override Annotation

An override annotation indicates that the annotated method is required to override a method in a super class. If a method with this annotation does not override its super-class’s method, the compiler will generate an error. Example 1 demonstrates the override annotation:

Java Annotation Example 1

What happens if a spelling mistake occurs with the method name? For example, if you change the name of the toString method to “tostring” and compile the code, you will get something like the following:

The Deprecated Annotation

This annotation indicates that when a deprecated program element is used, the compiler should warn you about it. Example 2 shows you the deprecated annotation.

Java Annotation Example 2

First, create a class with the deprecated method as follows:

Next, try to invoke this method from another class:

The doSomething() method in this example is declared as a deprecated method. Therefore, this method should not be used when this class is instantiated by other classes. If you compile Test_Deprecated.java, no warning messages will be generated by the compiler. But, if you try to compile TestAnnotations.java where the deprecated method is used, you will see something like this:

The Suppresswarnings Annotation

This annotation indicates that compiler warnings should be shielded in the annotated element and all of its sub-elements. The set of warnings suppressed in an element is the superset of the warnings in all of its containing sub-elements. As an example, if you annotate a class to suppress one warning and one of its methods to suppress another warning, both warnings will be suppressed at the method level only. See Example 3 for the suppresswarnings annotation.

Java Annotation Example 3

In this example, you are suppressing the deprecation warning for the method listing shown in Example 2. Because the method is suppressed, you are unlikely to view the “deprecation” warning any more.

Note: It is a good idea to use this annotation at the most deeply nested element where it is effective. Therefore, if you want to suppress a warning in a particular method, you should annotate that method rather than its class.

Meta-Annotations (Java Annotation Types)

Meta-annotations, which are actually known as the annotations of annotations, contain four types. These are:

The Target Annotation

The target annotation indicates the targeted elements of a class in which the annotation type will be applicable. It contains the following enumerated types as its value:

  • @Target(ElementType.TYPE) —can be applied to any element of a class
  • @Target(ElementType.FIELD) —can be applied to a field or property
  • @Target(ElementType.METHOD) —can be applied to a method level annotation
  • @Target(ElementType.PARAMETER) —can be applied to the parameters of a method
  • @Target(ElementType.CONSTRUCTOR) —can be applied to constructors
  • @Target(ElementType.LOCAL_VARIABLE) —can be applied to local variables
  • @Target(ElementType.ANNOTATION_TYPE) —indicates that the declared type itself is an annotation type

Example 4 demonstrates the target annotation:

Java Annotation Example 4

First, define an annotation named Test_Target with @Target metadata, as follows:

Next, create a class that will use the Test_Target annotation:

The @Target(ElementType.METHOD) indicates that this annotation type can be used to annotate only at the method levels. If you compile the preceding code, no warning messages will be shown. Now, if you declare a String variable and apply your newly created annotation, what will happen? Let me demonstrate this as follows:

The only change you can see from above is that the annotation declaration is shifted from method-level to field-level, which is not correct. Because you have defined your annotation @Test_Target to be applicable only at method-level, if you try to compile this class, you are likely to get something like this:

The Retention Annotation

The retention annotation indicates where and how long annotations with this type are to be retained. There are three values:

  • RetentionPolicy.SOURCE —Annotations with this type will be by retained only at the source level and will be ignored by the compiler
  • RetentionPolicy.CLASS —Annotations with this type will be by retained by the compiler at compile time, but will be ignored by the VM
  • RetentionPolicy.RUNTIME —Annotations with this type will be retained by the VM so they can be read only at run-time

Example 5 shows you the RetentionPolicy.RUNTIME value in action:

Java Annotation Example 5

In this example, the @Retention(RetentionPolicy.RUNTIME) annotation indicates that your Test_Retention annotation is to be retained by the VM so that it can be read reflectively at run-time.

The Documented Annotation

The documented annotation indicates that an annotation with this type should be documented by the javadoc tool. By default, annotations are not included in javadoc. But if @Documented is used, it then will be processed by javadoc-like tools and the annotation type information will also be included in the generated document. Example 6 demonstrates using @Documented further:

Java Annotation Example 6

Next, update your TestAnnotations class as follows:

Now, if you run the javadoc command and view the generated TestAnnotations.html file, you will see something like Figure 1.

As you can see from the screenshot, there is no annotation-type information for the doSomeTestRetention() method. But, this description is provided for the doSomeTestDocumented() method. This is because of the @Documented tag attached with your Test_Documented annotation. Your earlier annotation Test_Retention did not include this tag.

The Inherited Annotation

This is a bit of a complex annotation type. It indicates that the annotated class with this type is automatically inherited. More specifically, if you define an annotation with the @Inherited tag, then annotate a class with your annotation, and finally extend the class in a subclass, all properties of the parent class will be inherited into its subclass. With Example 7, you will get an idea about the benefits of using the @Inherited tag.

Java Annotation Example 7

First, define your annotation:

Next, annotate a class with your annotation:

As you can see, you do not have to define the interface methods inside the implementing class. These are automatically inherited because of using the @Inherited tag. What would happen if you define the implementing class in old-fashioned Java-style? Take a look at this—defining the implementation class in an old-style-java way:

Do you see the difference? You can see that you will have to implement all the methods that the parent interface owns. Besides the isInherited() and doSomething() methods from myParentObject, you will have to implement the equals(), toString(), and hasCode() methods of java.lang.Object and also the annotationType() method of java.lang.annotation.Annotation class. It does not matter whether you want to implement these methods or not; you will have to include these in your inherited object.

This article has shown you how you can make your development easier through the use of JDK5’s annotation feature. Annotations do not directly affect the semantics of a program. Development and deployment tools can read these annotations and process them in some fashion, perhaps producing additional Java programming language source files, XML documents, or other artifacts to be used in conjunction with the program containing the annotations. You now can do the same things as you would do before, but with less code and better compile-time error detection. The objective is to spend less time on unhandy code-writing and focus more on business logic rules. This article is Part I of a series on Java Annotations. In Part II, you will learn about how to use annotations to develop a simple Web application with a flat table. Lastly, in Part III you will see a complex example that includes multiple database tables relationships.

Java Annotation Resources

  • New Features and Enhancements J2SE 5.0 at http://www.mathcs.carleton.edu/courses/course_resources/j2se-1.5.0-docs/relnotes/features.html
  • J2SE 5.0 in a Nutshell at http://java.sun.com/developer/technicalArticles/releases/j2se15/
  • Annotations in Tiger, Part 1: Add metadata to Java code at http://www-128.ibm.com/developerworks/java/library/j-annotate1/
  • What’s New in Java 1.5? at http://www.cs.indiana.edu/classes/jett/sstamm/
  • Tiger in NetBeans4 at http://cropcrusher.web.infoseek.co.jp/shufujava/sunone/nb/nb4tiger_en.html
  • Taming the tiger at http://www.denverjug.org/meetings/files/200408_Tiger.pdf

About the Author

M. M. Islam Chisty currently works as a Software Engineer for M&H Informatics. His responsibilities include development and re-engineering of Web-based applications using J2EE technologies. M. Chisty has a B.S. degree in computer science from North South University. He loves spending his free time doing R&D on latest Java based technologies, watching TV, and listening to music. And, he’s a great WWE fan, too. You can contact M.Chisty at [email protected] .

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

What is the role of a project manager in software development, how to use optional in java, overview of the jad methodology, microsoft project tips and tricks, how to become a project manager in 2023, related stories, understanding types of thread synchronization errors in java, understanding memory consistency in java threads.

Developer.com

We are getting the warning

because of the following statement.

This is because we haven't defined the generic type of the array list. We can fix this warning by specifying generics inside angle brackets <> .

2. Type annotations

Before Java 8, annotations could be applied to declarations only. Now, type annotations can be used as well. This means that we can place annotations wherever we use a type.

Constructor invocations

Type definitions

This declaration specifies non-null variable str of type String to avoid NullPointerException .

This declaration specifies a non-null list of type String .

This declaration specifies a list of non-null values of type String .

extends and implements clause

throws clause

Type annotations enable Java code to be analyzed better and provide even stronger type checks.

Types of Annotations

1. Predefined annotations

  • @Deprecated
  • @SuppressWarnings
  • @SafeVarargs
  • @FunctionalInterface

2. Meta-annotations

  • @Documented
  • @Repeatable

3. Custom annotations

These annotation types are described in detail in the Java Annotation Types tutorial.

  • Use of Annotations
  • Compiler instructions - Annotations can be used for giving instructions to the compiler, detect errors or suppress warnings. The built-in annotations @Deprecated , @Override , @SuppressWarnings are used for these purposes.
  • Compile-time instructions - Compile-time instructions provided by these annotations help the software build tools to generate code, XML files and many more.
  • Runtime instructions - Some annotations can be defined to give instructions to the program at runtime. These annotations are accessed using Java Reflection.

Table of Contents

  • Types of Java Annotations

Sorry about that.

Related Tutorials

Java Tutorial

Javatpoint Logo

Java New Features

Java 9 features, java 8 features, java 7 features, java 4/5 features.

JavaTpoint

Example to specify annoation for a class

Example to specify annotation for a class, methods or fields.

@Retention annotation is used to specify to what level annotation will be available.

Example to specify the RetentionPolicy

Example of custom annotation: creating, applying and accessing annotation.

Let's see the simple example of creating, applying and accessing annotation.

File: Test.java

How built-in annotaions are used in real scenario?

In real scenario, java programmer only need to apply annotation. He/She doesn't need to create and access annotation. Creating and Accessing annotation is performed by the implementation provider. On behalf of the annotation, java compiler or JVM performs some additional operations.

By default, annotations are not inherited to subclasses. The @Inherited annotation marks the annotation to be inherited to subclasses.

@Documented

The @Documented Marks the annotation for inclusion in the documentation.

Youtube

  • Send your Feedback to [email protected]

Help Others, Please Share

facebook

Learn Latest Tutorials

Splunk tutorial

Transact-SQL

Tumblr tutorial

Reinforcement Learning

R Programming tutorial

R Programming

RxJS tutorial

React Native

Python Design Patterns

Python Design Patterns

Python Pillow tutorial

Python Pillow

Python Turtle tutorial

Python Turtle

Keras tutorial

Preparation

Aptitude

Verbal Ability

Interview Questions

Interview Questions

Company Interview Questions

Company Questions

Trending Technologies

Artificial Intelligence

Artificial Intelligence

AWS Tutorial

Cloud Computing

Hadoop tutorial

Data Science

Angular 7 Tutorial

Machine Learning

DevOps Tutorial

B.Tech / MCA

DBMS tutorial

Data Structures

DAA tutorial

Operating System

Computer Network tutorial

Computer Network

Compiler Design tutorial

Compiler Design

Computer Organization and Architecture

Computer Organization

Discrete Mathematics Tutorial

Discrete Mathematics

Ethical Hacking

Ethical Hacking

Computer Graphics Tutorial

Computer Graphics

Software Engineering

Software Engineering

html tutorial

Web Technology

Cyber Security tutorial

Cyber Security

Automata Tutorial

C Programming

C++ tutorial

Control System

Data Mining Tutorial

Data Mining

Data Warehouse Tutorial

Data Warehouse

RSS Feed

  • Python »
  • 3.12.2 Documentation »
  • Python HOWTOs »
  • Annotations Best Practices
  • Theme Auto Light Dark |

Annotations Best Practices ¶

Larry Hastings

Accessing The Annotations Dict Of An Object In Python 3.10 And Newer ¶

Python 3.10 adds a new function to the standard library: inspect.get_annotations() . In Python versions 3.10 and newer, calling this function is the best practice for accessing the annotations dict of any object that supports annotations. This function can also “un-stringize” stringized annotations for you.

If for some reason inspect.get_annotations() isn’t viable for your use case, you may access the __annotations__ data member manually. Best practice for this changed in Python 3.10 as well: as of Python 3.10, o.__annotations__ is guaranteed to always work on Python functions, classes, and modules. If you’re certain the object you’re examining is one of these three specific objects, you may simply use o.__annotations__ to get at the object’s annotations dict.

However, other types of callables–for example, callables created by functools.partial() –may not have an __annotations__ attribute defined. When accessing the __annotations__ of a possibly unknown object, best practice in Python versions 3.10 and newer is to call getattr() with three arguments, for example getattr(o, '__annotations__', None) .

Before Python 3.10, accessing __annotations__ on a class that defines no annotations but that has a parent class with annotations would return the parent’s __annotations__ . In Python 3.10 and newer, the child class’s annotations will be an empty dict instead.

Accessing The Annotations Dict Of An Object In Python 3.9 And Older ¶

In Python 3.9 and older, accessing the annotations dict of an object is much more complicated than in newer versions. The problem is a design flaw in these older versions of Python, specifically to do with class annotations.

Best practice for accessing the annotations dict of other objects–functions, other callables, and modules–is the same as best practice for 3.10, assuming you aren’t calling inspect.get_annotations() : you should use three-argument getattr() to access the object’s __annotations__ attribute.

Unfortunately, this isn’t best practice for classes. The problem is that, since __annotations__ is optional on classes, and because classes can inherit attributes from their base classes, accessing the __annotations__ attribute of a class may inadvertently return the annotations dict of a base class. As an example:

This will print the annotations dict from Base , not Derived .

Your code will have to have a separate code path if the object you’re examining is a class ( isinstance(o, type) ). In that case, best practice relies on an implementation detail of Python 3.9 and before: if a class has annotations defined, they are stored in the class’s __dict__ dictionary. Since the class may or may not have annotations defined, best practice is to call the get method on the class dict.

To put it all together, here is some sample code that safely accesses the __annotations__ attribute on an arbitrary object in Python 3.9 and before:

After running this code, ann should be either a dictionary or None . You’re encouraged to double-check the type of ann using isinstance() before further examination.

Note that some exotic or malformed type objects may not have a __dict__ attribute, so for extra safety you may also wish to use getattr() to access __dict__ .

Manually Un-Stringizing Stringized Annotations ¶

In situations where some annotations may be “stringized”, and you wish to evaluate those strings to produce the Python values they represent, it really is best to call inspect.get_annotations() to do this work for you.

If you’re using Python 3.9 or older, or if for some reason you can’t use inspect.get_annotations() , you’ll need to duplicate its logic. You’re encouraged to examine the implementation of inspect.get_annotations() in the current Python version and follow a similar approach.

In a nutshell, if you wish to evaluate a stringized annotation on an arbitrary object o :

If o is a module, use o.__dict__ as the globals when calling eval() .

If o is a class, use sys.modules[o.__module__].__dict__ as the globals , and dict(vars(o)) as the locals , when calling eval() .

If o is a wrapped callable using functools.update_wrapper() , functools.wraps() , or functools.partial() , iteratively unwrap it by accessing either o.__wrapped__ or o.func as appropriate, until you have found the root unwrapped function.

If o is a callable (but not a class), use o.__globals__ as the globals when calling eval() .

However, not all string values used as annotations can be successfully turned into Python values by eval() . String values could theoretically contain any valid string, and in practice there are valid use cases for type hints that require annotating with string values that specifically can’t be evaluated. For example:

PEP 604 union types using | , before support for this was added to Python 3.10.

Definitions that aren’t needed at runtime, only imported when typing.TYPE_CHECKING is true.

If eval() attempts to evaluate such values, it will fail and raise an exception. So, when designing a library API that works with annotations, it’s recommended to only attempt to evaluate string values when explicitly requested to by the caller.

Best Practices For __annotations__ In Any Python Version ¶

You should avoid assigning to the __annotations__ member of objects directly. Let Python manage setting __annotations__ .

If you do assign directly to the __annotations__ member of an object, you should always set it to a dict object.

If you directly access the __annotations__ member of an object, you should ensure that it’s a dictionary before attempting to examine its contents.

You should avoid modifying __annotations__ dicts.

You should avoid deleting the __annotations__ attribute of an object.

__annotations__ Quirks ¶

In all versions of Python 3, function objects lazy-create an annotations dict if no annotations are defined on that object. You can delete the __annotations__ attribute using del fn.__annotations__ , but if you then access fn.__annotations__ the object will create a new empty dict that it will store and return as its annotations. Deleting the annotations on a function before it has lazily created its annotations dict will throw an AttributeError ; using del fn.__annotations__ twice in a row is guaranteed to always throw an AttributeError .

Everything in the above paragraph also applies to class and module objects in Python 3.10 and newer.

In all versions of Python 3, you can set __annotations__ on a function object to None . However, subsequently accessing the annotations on that object using fn.__annotations__ will lazy-create an empty dictionary as per the first paragraph of this section. This is not true of modules and classes, in any Python version; those objects permit setting __annotations__ to any Python value, and will retain whatever value is set.

If Python stringizes your annotations for you (using from __future__ import annotations ), and you specify a string as an annotation, the string will itself be quoted. In effect the annotation is quoted twice. For example:

This prints {'a': "'str'"} . This shouldn’t really be considered a “quirk”; it’s mentioned here simply because it might be surprising.

Table of Contents

  • Accessing The Annotations Dict Of An Object In Python 3.10 And Newer
  • Accessing The Annotations Dict Of An Object In Python 3.9 And Older
  • Manually Un-Stringizing Stringized Annotations
  • Best Practices For __annotations__ In Any Python Version
  • __annotations__ Quirks

Previous topic

Python support for the Linux perf profiler

Isolating Extension Modules

  • Report a Bug
  • Show Source

Writers' Center

Eastern Washington University

Reading and Study Strategies

What is annotating and why do it, annotation explained, steps to annotating a source, annotating strategies.

  • Using a Dictionary
  • Study Skills

[ Back to resource home ]

An image of writing consultants meeting with students.

[email protected] 509.359.2779

Cheney Campus   JFK Library Learning Commons

Spokane Campus Catalyst Building C451 and C452

Stay Connected! Instagram  Facebook

Helpful Links

Software for Annotating

ProQuest Flow (sign up with your EWU email)

FoxIt PDF Reader

Adobe Reader Pro  - available on all campus computers

Track Changes in Microsoft Word

What is Annotating?

Annotating is any action that deliberately interacts with a text to enhance the reader's understanding of, recall of, and reaction to the text. Sometimes called "close reading," annotating usually involves highlighting or underlining key pieces of text and making notes in the margins of the text. This page will introduce you to several effective strategies for annotating a text that will help you get the most out of your reading.

Why Annotate?

By annotating a text, you will ensure that you understand what is happening in a text after you've read it. As you annotate, you should note the author's main points, shifts in the message or perspective of the text, key areas of focus, and your own thoughts as you read. However, annotating isn't just for people who feel challenged when reading academic texts. Even if you regularly understand and remember what you read, annotating will help you summarize a text, highlight important pieces of information, and ultimately prepare yourself for discussion and writing prompts that your instructor may give you. Annotating means you are doing the hard work while you read, allowing you to reference your previous work and have a clear jumping-off point for future work.

1. Survey : This is your first time through the reading

You can annotate by hand or by using document software. You can also annotate on post-its if you have a text you do not want to mark up. As you annotate, use these strategies to make the most of your efforts:

  • Include a key or legend on your paper that indicates what each marking is for, and use a different marking for each type of information. Example: Underline for key points, highlight for vocabulary, and circle for transition points.
  • If you use highlighters, consider using different colors for different types of reactions to the text. Example: Yellow for definitions, orange for questions, and blue for disagreement/confusion.
  • Dedicate different tasks to each margin: Use one margin to make an outline of the text (thesis statement, description, definition #1, counter argument, etc.) and summarize main ideas, and use the other margin to note your thoughts, questions, and reactions to the text.

Lastly, as you annotate, make sure you are including descriptions of the text as well as your own reactions to the text. This will allow you to skim your notations at a later date to locate key information and quotations, and to recall your thought processes more easily and quickly.

  • Next: Using a Dictionary >>
  • Last Updated: Jul 21, 2021 3:01 PM
  • URL: https://research.ewu.edu/writers_c_read_study_strategies
  • Trending Now
  • Data Structures & Algorithms
  • Foundational Courses
  • Data Science
  • Practice Problem
  • Machine Learning
  • System Design
  • DevOps Tutorial
  • Web Development
  • Web Browser

Related Articles

  • Given two arrays, find n+m-1 unique sum pairs
  • Simple Multithreaded Download Manager in Python
  • Count the number of rows and columns of Pandas dataframe
  • What does the if __name__ == “__main__”: do?
  • Python Virtual Environment | Introduction
  • Python - Multi-Line Statements
  • How to Use Google Bard with Python
  • Maximum Depth or Height Of a Binary Tree with python
  • Inbuilt Data Structures in Python
  • Decimal Functions in Python | Set 2 (logical_and(), normalize(), quantize(), rotate() ... )
  • Decimal Functions in Python | Set 1
  • wxPython | Exit() function in wxPython
  • Python Set Exercise
  • Python GUI - tkinter
  • Iterator Functions in Python | Set 2 (islice(), starmap(), tee()..)
  • Iterator Functions in Python | Set 1
  • Youtube Data API Subscription | Set-2
  • Statistical Functions in Python | Set 2 ( Measure of Spread)
  • Coroutine in Python

Function Annotations in Python

Basic Terminology

What are Function annotations?

  • Python supports dynamic typing and hence no module is provided for type checking. Annotations like [def foo(a:”int”, b:”float”=5.0) -> ”int”] (syntax described in detail in the next section) can be used to collect information about the type of the parameters and the return type of the function to keep track of the type change occurring in the function. ‘mypy’ is one such library.
  • String based annotations can be used by the libraries to provide better help messages at compile time regarding the functionalities of various methods, classes and modules.

Syntax of function annotations

  • Annotations for simple parameters : The argument name is followed by ‘:’ which is then followed by the expression. Annotation syntax is shown below. def foobar(a: expression, b: expression = 5):
  • Annotations for excess parameters : Excess parameters for e.g. *args and **kwargs, allow arbitrary number of arguments to be passed in a function call. Annotation syntax of such parameters is shown below. def foobar(*args: expression, *kwargs: expression):
  • Annotations for nested parameters : Nested parameters are useful feature of python 2x where a tuple is passed in a function call and automatic unpacking takes place. This feature is removed in python 3x and manual unpacking should be done. Annotation is done after the variable and not after the tuple as shown below. def foobar((a: expression, b: expression), (c: expression, d: expression)):
  • Annotations for return type : Annotating return type is slightly different from annotating function arguments. The ‘->’ is followed by expression which is further followed by ‘:’. Annotation syntax of return type is shown below. def foobar(a: expression)->expression:

an annotation method

Accessing Function Annotations

an annotation method

Application of Function Annotations

an annotation method

Please Login to comment...

  • shubham_singh

Improve your Coding Skills with Practice

 alt=

What kind of Experience do you want to share?

Learning Center

Annotating Texts

What is annotation.

Annotation can be:

  • A systematic summary of the text that you create within the document
  • A key tool for close reading that helps you uncover patterns, notice important words, and identify main points
  • An active learning strategy that improves comprehension and retention of information

Why annotate?

  • Isolate and organize important material
  • Identify key concepts
  • Monitor your learning as you read
  • Make exam prep effective and streamlined
  • Can be more efficient than creating a separate set of reading notes

How do you annotate?

Summarize key points in your own words .

  • Use headers and words in bold to guide you
  • Look for main ideas, arguments, and points of evidence
  • Notice how the text organizes itself. Chronological order? Idea trees? Etc.

Circle key concepts and phrases

  • What words would it be helpful to look-up at the end?
  • What terms show up in lecture? When are different words used for similar concepts? Why?

Write brief comments and questions in the margins

  • Be as specific or broad as you would like—use these questions to activate your thinking about the content
  • See our handout on reading comprehension tips for some examples

Use abbreviations and symbols

  • Try ? when you have a question or something you need to explore further
  • Try ! When something is interesting, a connection, or otherwise worthy of note
  • Try * For anything that you might use as an example or evidence when you use this information.
  • Ask yourself what other system of symbols would make sense to you.

Highlight/underline

  • Highlight or underline, but mindfully. Check out our resource on strategic highlighting for tips on when and how to highlight.

Use comment and highlight features built into pdfs, online/digital textbooks, or other apps and browser add-ons

  • Are you using a pdf? Explore its highlight, edit, and comment functions to support your annotations
  • Some browsers have add-ons or extensions that allow you to annotate web pages or web-based documents
  • Does your digital or online textbook come with an annotation feature?
  • Can your digital text be imported into a note-taking tool like OneNote, EverNote, or Google Keep? If so, you might be able to annotate texts in those apps

What are the most important takeaways?

  • Annotation is about increasing your engagement with a text
  • Increased engagement, where you think about and process the material then expand on your learning, is how you achieve mastery in a subject
  • As you annotate a text, ask yourself: how would I explain this to a friend?
  • Put things in your own words and draw connections to what you know and wonder

The table below demonstrates this process using a geography textbook excerpt (Press 2004):

A chart featuring a passage from a text in the left column and then columns that illustrate annotations that include too much writing, not enough writing, and a good balance of writing.

A common concern about annotating texts: It takes time!

Yes, it can, but that time isn’t lost—it’s invested.

Spending the time to annotate on the front end does two important things:

  • It saves you time later when you’re studying. Your annotated notes will help speed up exam prep, because you can review critical concepts quickly and efficiently.
  • It increases the likelihood that you will retain the information after the course is completed. This is especially important when you are supplying the building blocks of your mind and future career.

One last tip: Try separating the reading and annotating processes! Quickly read through a section of the text first, then go back and annotate.

Works consulted:

Nist, S., & Holschuh, J. (2000). Active learning: strategies for college success. Boston: Allyn and Bacon. 202-218.

Simpson, M., & Nist, S. (1990). Textbook annotation: An effective and efficient study strategy for college students. Journal of Reading, 34: 122-129.

Press, F. (2004). Understanding earth (4th ed). New York: W.H. Freeman. 208-210.

Creative Commons License

Make a Gift

an annotation method

How to Annotate Texts

Use the links below to jump directly to any section of this guide:

Annotation Fundamentals

How to start annotating , how to annotate digital texts, how to annotate a textbook, how to annotate a scholarly article or book, how to annotate literature, how to annotate images, videos, and performances, additional resources for teachers.

Writing in your books can make you smarter. Or, at least (according to education experts), annotation–an umbrella term for underlining, highlighting, circling, and, most importantly, leaving comments in the margins–helps students to remember and comprehend what they read. Annotation is like a conversation between reader and text. Proper annotation allows students to record their own opinions and reactions, which can serve as the inspiration for research questions and theses. So, whether you're reading a novel, poem, news article, or science textbook, taking notes along the way can give you an advantage in preparing for tests or writing essays. This guide contains resources that explain the benefits of annotating texts, provide annotation tools, and suggest approaches for diverse kinds of texts; the last section includes lesson plans and exercises for teachers.

Why annotate? As the resources below explain, annotation allows students to emphasize connections to material covered elsewhere in the text (or in other texts), material covered previously in the course, or material covered in lectures and discussion. In other words, proper annotation is an organizing tool and a time saver. The links in this section will introduce you to the theory, practice, and purpose of annotation. 

How to Mark a Book, by Mortimer Adler

This famous, charming essay lays out the case for marking up books, and provides practical suggestions at the end including underlining, highlighting, circling key words, using vertical lines to mark shifts in tone/subject, numbering points in an argument, and keeping track of questions that occur to you as you read. 

How Annotation Reshapes Student Thinking (TeacherHUB)

In this article, a high school teacher discusses the importance of annotation and how annotation encourages more effective critical thinking.

The Future of Annotation (Journal of Business and Technical Communication)

This scholarly article summarizes research on the benefits of annotation in the classroom and in business. It also discusses how technology and digital texts might affect the future of annotation. 

Annotating to Deepen Understanding (Texas Education Agency)

This website provides another introduction to annotation (designed for 11th graders). It includes a helpful section that teaches students how to annotate reading comprehension passages on tests.

Once you understand what annotation is, you're ready to begin. But what tools do you need? How do you prepare? The resources linked in this section list strategies and techniques you can use to start annotating. 

What is Annotating? (Charleston County School District)

This resource gives an overview of annotation styles, including useful shorthands and symbols. This is a good place for a student who has never annotated before to begin.

How to Annotate Text While Reading (YouTube)

This video tutorial (appropriate for grades 6–10) explains the basic ins and outs of annotation and gives examples of the type of information students should be looking for.

Annotation Practices: Reading a Play-text vs. Watching Film (U Calgary)

This blog post, written by a student, talks about how the goals and approaches of annotation might change depending on the type of text or performance being observed. 

Annotating Texts with Sticky Notes (Lyndhurst Schools)

Sometimes students are asked to annotate books they don't own or can't write in for other reasons. This resource provides some strategies for using sticky notes instead.

Teaching Students to Close Read...When You Can't Mark the Text (Performing in Education)

Here, a sixth grade teacher demonstrates the strategies she uses for getting her students to annotate with sticky notes. This resource includes a link to the teacher's free Annotation Bookmark (via Teachers Pay Teachers).

Digital texts can present a special challenge when it comes to annotation; emerging research suggests that many students struggle to critically read and retain information from digital texts. However, proper annotation can solve the problem. This section contains links to the most highly-utilized platforms for electronic annotation.

Evernote is one of the two big players in the "digital annotation apps" game. In addition to allowing users to annotate digital documents, the service (for a fee) allows users to group multiple formats (PDF, webpages, scanned hand-written notes) into separate notebooks, create voice recordings, and sync across all sorts of devices. 

OneNote is Evernote's main competitor. Reviews suggest that OneNote allows for more freedom for digital note-taking than Evernote, but that it is slightly more awkward to import and annotate a PDF, especially on certain platforms. However, OneNote's free version is slightly more feature-filled, and OneNote allows you to link your notes to time stamps on an audio recording.

Diigo is a basic browser extension that allows a user to annotate webpages. Diigo also offers a Screenshot app that allows for direct saving to Google Drive.

While the creators of Hypothesis like to focus on their app's social dimension, students are more likely to be interested in the private highlighting and annotating functions of this program.

Foxit PDF Reader

Foxit is one of the leading PDF readers. Though the full suite must be purchased, Foxit offers a number of annotation and highlighting tools for free.

Nitro PDF Reader

This is another well-reviewed, free PDF reader that includes annotation and highlighting. Annotation, text editing, and other tools are included in the free version.

Goodreader is a very popular Mac-only app that includes annotation and editing tools for PDFs, Word documents, Powerpoint, and other formats.

Although textbooks have vocabulary lists, summaries, and other features to emphasize important material, annotation can allow students to process information and discover their own connections. This section links to guides and video tutorials that introduce you to textbook annotation. 

Annotating Textbooks (Niagara University)

This PDF provides a basic introduction as well as strategies including focusing on main ideas, working by section or chapter, annotating in your own words, and turning section headings into questions.

A Simple Guide to Text Annotation (Catawba College)

The simple, practical strategies laid out in this step-by-step guide will help students learn how to break down chapters in their textbooks using main ideas, definitions, lists, summaries, and potential test questions.

Annotating (Mercer Community College)

This packet, an excerpt from a literature textbook, provides a short exercise and some examples of how to do textbook annotation, including using shorthand and symbols.

Reading Your Healthcare Textbook: Annotation (Saddleback College)

This powerpoint contains a number of helpful suggestions, especially for students who are new to annotation. It emphasizes limited highlighting, lots of student writing, and using key words to find the most important information in a textbook. Despite the title, it is useful to a student in any discipline.

Annotating a Textbook (Excelsior College OWL)

This video (with included transcript) discusses how to use textbook features like boxes and sidebars to help guide annotation. It's an extremely helpful, detailed discussion of how textbooks are organized.

Because scholarly articles and books have complex arguments and often depend on technical vocabulary, they present particular challenges for an annotating student. The resources in this section help students get to the heart of scholarly texts in order to annotate and, by extension, understand the reading.

Annotating a Text (Hunter College)

This resource is designed for college students and shows how to annotate a scholarly article using highlighting, paraphrase, a descriptive outline, and a two-margin approach. It ends with a sample passage marked up using the strategies provided. 

Guide to Annotating the Scholarly Article (ReadWriteThink.org)

This is an effective introduction to annotating scholarly articles across all disciplines. This resource encourages students to break down how the article uses primary and secondary sources and to annotate the types of arguments and persuasive strategies (synthesis, analysis, compare/contrast).

How to Highlight and Annotate Your Research Articles (CHHS Media Center)

This video, developed by a high school media specialist, provides an effective beginner-level introduction to annotating research articles. 

How to Read a Scholarly Book (AndrewJacobs.org)

In this essay, a college professor lets readers in on the secrets of scholarly monographs. Though he does not discuss annotation, he explains how to find a scholarly book's thesis, methodology, and often even a brief literature review in the introduction. This is a key place for students to focus when creating annotations. 

A 5-step Approach to Reading Scholarly Literature and Taking Notes (Heather Young Leslie)

This resource, written by a professor of anthropology, is an even more comprehensive and detailed guide to reading scholarly literature. Combining the annotation techniques above with the reading strategy here allows students to process scholarly book efficiently. 

Annotation is also an important part of close reading works of literature. Annotating helps students recognize symbolism, double meanings, and other literary devices. These resources provide additional guidelines on annotating literature.

AP English Language Annotation Guide (YouTube)

In this ~10 minute video, an AP Language teacher provides tips and suggestions for using annotations to point out rhetorical strategies and other important information.

Annotating Text Lesson (YouTube)

In this video tutorial, an English teacher shows how she uses the white board to guide students through annotation and close reading. This resource uses an in-depth example to model annotation step-by-step.

Close Reading a Text and Avoiding Pitfalls (Purdue OWL)

This resources demonstrates how annotation is a central part of a solid close reading strategy; it also lists common mistakes to avoid in the annotation process.

AP Literature Assignment: Annotating Literature (Mount Notre Dame H.S.)

This brief assignment sheet contains suggestions for what to annotate in a novel, including building connections between parts of the book, among multiple books you are reading/have read, and between the book and your own experience. It also includes samples of quality annotations.

AP Handout: Annotation Guide (Covington Catholic H.S.)

This annotation guide shows how to keep track of symbolism, figurative language, and other devices in a novel using a highlighter, a pencil, and every part of a book (including the front and back covers).

In addition to written resources, it's possible to annotate visual "texts" like theatrical performances, movies, sculptures, and paintings. Taking notes on visual texts allows students to recall details after viewing a resource which, unlike a book, can't be re-read or re-visited ( for example, a play that has finished its run, or an art exhibition that is far away). These resources draw attention to the special questions and techniques that students should use when dealing with visual texts.

How to Take Notes on Videos (U of Southern California)

This resource is a good place to start for a student who has never had to take notes on film before. It briefly outlines three general approaches to note-taking on a film. 

How to Analyze a Movie, Step-by-Step (San Diego Film Festival)

This detailed guide provides lots of tips for film criticism and analysis. It contains a list of specific questions to ask with respect to plot, character development, direction, musical score, cinematography, special effects, and more. 

How to "Read" a Film (UPenn)

This resource provides an academic perspective on the art of annotating and analyzing a film. Like other resources, it provides students a checklist of things to watch out for as they watch the film.

Art Annotation Guide (Gosford Hill School)

This resource focuses on how to annotate a piece of art with respect to its formal elements like line, tone, mood, and composition. It contains a number of helpful questions and relevant examples. 

Photography Annotation (Arts at Trinity)

This resource is designed specifically for photography students. Like some of the other resources on this list, it primarily focuses on formal elements, but also shows students how to integrate the specific technical vocabulary of modern photography. This resource also contains a number of helpful sample annotations.

How to Review a Play (U of Wisconsin)

This resource from the University of Wisconsin Writing Center is designed to help students write a review of a play. It contains suggested questions for students to keep in mind as they watch a given production. This resource helps students think about staging, props, script alterations, and many other key elements of a performance.

This section contains links to lessons plans and exercises suitable for high school and college instructors.

Beyond the Yellow Highlighter: Teaching Annotation Skills to Improve Reading Comprehension (English Journal)

In this journal article, a high school teacher talks about her approach to teaching annotation. This article makes a clear distinction between annotation and mere highlighting.

Lesson Plan for Teaching Annotation, Grades 9–12 (readwritethink.org)

This lesson plan, published by the National Council of Teachers of English, contains four complete lessons that help introduce high school students to annotation.

Teaching Theme Using Close Reading (Performing in Education)

This lesson plan was developed by a middle school teacher, and is aligned to Common Core. The teacher presents her strategies and resources in comprehensive fashion.

Analyzing a Speech Using Annotation (UNC-TV/PBS Learning Media)

This complete lesson plan, which includes a guide for the teacher and relevant handouts for students, will prepare students to analyze both the written and presentation components of a speech. This lesson plan is best for students in 6th–10th grade.

Writing to Learn History: Annotation and Mini-Writes (teachinghistory.org)

This teaching guide, developed for high school History classes, provides handouts and suggested exercises that can help students become more comfortable with annotating historical sources.

Writing About Art (The College Board)

This Prezi presentation is useful to any teacher introducing students to the basics of annotating art. The presentation covers annotating for both formal elements and historical/cultural significance.

Film Study Worksheets (TeachWithMovies.org)

This resource contains links to a general film study worksheet, as well as specific worksheets for novel adaptations, historical films, documentaries, and more. These resources are appropriate for advanced middle school students and some high school students. 

Annotation Practice Worksheet (La Guardia Community College)

This worksheet has a sample text and instructions for students to annotate it. It is a useful resource for teachers who want to give their students a chance to practice, but don't have the time to select an appropriate piece of text. 

  • PDFs for all 136 Lit Terms we cover
  • Downloads of 1871 LitCharts Lit Guides
  • Teacher Editions for every Lit Guide
  • Explanations and citation info for 39,325 quotes across 1871 books
  • Downloadable (PDF) line-by-line translations of every Shakespeare play

Need something? Request a new guide .

How can we improve? Share feedback .

LitCharts is hiring!

The LitCharts.com logo.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base
  • Citing sources
  • What Is an Annotated Bibliography? | Examples & Format

What Is an Annotated Bibliography? | Examples & Format

Published on March 9, 2021 by Jack Caulfield . Revised on August 23, 2022.

An annotated bibliography is a list of source references that includes a short descriptive text (an annotation) for each source. It may be assigned as part of the research process for a paper , or as an individual assignment to gather and read relevant sources on a topic.

Scribbr’s free Citation Generator allows you to easily create and manage your annotated bibliography in APA or MLA style. To generate a perfectly formatted annotated bibliography, select the source type, fill out the relevant fields, and add your annotation.

An example of an annotated source is shown below:

Annotated source example

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

upload-your-document-ai-proofreader

Table of contents

Annotated bibliography format: apa, mla, chicago, how to write an annotated bibliography, descriptive annotation example, evaluative annotation example, reflective annotation example, finding sources for your annotated bibliography, frequently asked questions about annotated bibliographies.

Make sure your annotated bibliography is formatted according to the guidelines of the style guide you’re working with. Three common styles are covered below:

In APA Style , both the reference entry and the annotation should be double-spaced and left-aligned.

The reference entry itself should have a hanging indent . The annotation follows on the next line, and the whole annotation should be indented to match the hanging indent. The first line of any additional paragraphs should be indented an additional time.

APA annotated bibliography

In an MLA style annotated bibliography , the Works Cited entry and the annotation are both double-spaced and left-aligned.

The Works Cited entry has a hanging indent. The annotation itself is indented 1 inch (twice as far as the hanging indent). If there are two or more paragraphs in the annotation, the first line of each paragraph is indented an additional half-inch, but not if there is only one paragraph.

MLA annotated bibliography

Chicago style

In a  Chicago style annotated bibliography , the bibliography entry itself should be single-spaced and feature a hanging indent.

The annotation should be indented, double-spaced, and left-aligned. The first line of any additional paragraphs should be indented an additional time.

Chicago annotated bibliography

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

For each source, start by writing (or generating ) a full reference entry that gives the author, title, date, and other information. The annotated bibliography format varies based on the citation style you’re using.

The annotations themselves are usually between 50 and 200 words in length, typically formatted as a single paragraph. This can vary depending on the word count of the assignment, the relative length and importance of different sources, and the number of sources you include.

Consider the instructions you’ve been given or consult your instructor to determine what kind of annotations they’re looking for:

  • Descriptive annotations : When the assignment is just about gathering and summarizing information, focus on the key arguments and methods of each source.
  • Evaluative annotations : When the assignment is about evaluating the sources , you should also assess the validity and effectiveness of these arguments and methods.
  • Reflective annotations : When the assignment is part of a larger research process, you need to consider the relevance and usefulness of the sources to your own research.

These specific terms won’t necessarily be used. The important thing is to understand the purpose of your assignment and pick the approach that matches it best. Interactive examples of the different styles of annotation are shown below.

A descriptive annotation summarizes the approach and arguments of a source in an objective way, without attempting to assess their validity.

In this way, it resembles an abstract , but you should never just copy text from a source’s abstract, as this would be considered plagiarism . You’ll naturally cover similar ground, but you should also consider whether the abstract omits any important points from the full text.

The interactive example shown below describes an article about the relationship between business regulations and CO 2 emissions.

Rieger, A. (2019). Doing business and increasing emissions? An exploratory analysis of the impact of business regulation on CO 2 emissions. Human Ecology Review , 25 (1), 69–86. https://www.jstor.org/stable/26964340

An evaluative annotation also describes the content of a source, but it goes on to evaluate elements like the validity of the source’s arguments and the appropriateness of its methods .

For example, the following annotation describes, and evaluates the effectiveness of, a book about the history of Western philosophy.

Kenny, A. (2010). A new history of Western philosophy: In four parts . Oxford University Press.

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

  • Missing commas and periods
  • Incorrect usage of “et al.”
  • Ampersands (&) in narrative citations
  • Missing reference entries

an annotation method

A reflective annotation is similar to an evaluative one, but it focuses on the source’s usefulness or relevance to your own research.

Reflective annotations are often required when the point is to gather sources for a future research project, or to assess how they were used in a project you already completed.

The annotation below assesses the usefulness of a particular article for the author’s own research in the field of media studies.

Manovich, Lev. (2009). The practice of everyday (media) life: From mass consumption to mass cultural production? Critical Inquiry , 35 (2), 319–331. https://www.jstor.org/stable/10.1086/596645

Manovich’s article assesses the shift from a consumption-based media culture (in which media content is produced by a small number of professionals and consumed by a mass audience) to a production-based media culture (in which this mass audience is just as active in producing content as in consuming it). He is skeptical of some of the claims made about this cultural shift; specifically, he argues that the shift towards user-made content must be regarded as more reliant upon commercial media production than it is typically acknowledged to be. However, he regards web 2.0 as an exciting ongoing development for art and media production, citing its innovation and unpredictability.

The article is outdated in certain ways (it dates from 2009, before the launch of Instagram, to give just one example). Nevertheless, its critical engagement with the possibilities opened up for media production by the growth of social media is valuable in a general sense, and its conceptualization of these changes frequently applies just as well to more current social media platforms as it does to Myspace. Conceptually, I intend to draw on this article in my own analysis of the social dynamics of Twitter and Instagram.

Before you can write your annotations, you’ll need to find sources . If the annotated bibliography is part of the research process for a paper, your sources will be those you consult and cite as you prepare the paper. Otherwise, your assignment and your choice of topic will guide you in what kind of sources to look for.

Make sure that you’ve clearly defined your topic , and then consider what keywords are relevant to it, including variants of the terms. Use these keywords to search databases (e.g., Google Scholar ), using Boolean operators to refine your search.

Sources can include journal articles, books, and other source types , depending on the scope of the assignment. Read the abstracts or blurbs of the sources you find to see whether they’re relevant, and try exploring their bibliographies to discover more. If a particular source keeps showing up, it’s probably important.

Once you’ve selected an appropriate range of sources, read through them, taking notes that you can use to build up your annotations. You may even prefer to write your annotations as you go, while each source is fresh in your mind.

An annotated bibliography is an assignment where you collect sources on a specific topic and write an annotation for each source. An annotation is a short text that describes and sometimes evaluates the source.

Any credible sources on your topic can be included in an annotated bibliography . The exact sources you cover will vary depending on the assignment, but you should usually focus on collecting journal articles and scholarly books . When in doubt, utilize the CRAAP test !

Each annotation in an annotated bibliography is usually between 50 and 200 words long. Longer annotations may be divided into paragraphs .

The content of the annotation varies according to your assignment. An annotation can be descriptive, meaning it just describes the source objectively; evaluative, meaning it assesses its usefulness; or reflective, meaning it explains how the source will be used in your own research .

A source annotation in an annotated bibliography fulfills a similar purpose to an abstract : they’re both intended to summarize the approach and key points of a source.

However, an annotation may also evaluate the source , discussing the validity and effectiveness of its arguments. Even if your annotation is purely descriptive , you may have a different perspective on the source from the author and highlight different key points.

You should never just copy text from the abstract for your annotation, as doing so constitutes plagiarism .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Caulfield, J. (2022, August 23). What Is an Annotated Bibliography? | Examples & Format. Scribbr. Retrieved February 22, 2024, from https://www.scribbr.com/citing-sources/annotated-bibliography/

Is this article helpful?

Jack Caulfield

Jack Caulfield

Other students also liked, evaluating sources | methods & examples, how to find sources | scholarly articles, books, etc., hanging indent | word & google docs instructions, what is your plagiarism score.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Published: 21 February 2024

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

  • Jens F. Tillmann   ORCID: orcid.org/0000-0002-3030-1571 1   na1 ,
  • Alexander I. Hsu   ORCID: orcid.org/0000-0002-8198-945X 2   na1 ,
  • Martin K. Schwarz   ORCID: orcid.org/0000-0002-5898-751X 1 &
  • Eric A. Yttri   ORCID: orcid.org/0000-0002-7214-1481 2 , 3  

Nature Methods ( 2024 ) Cite this article

81 Altmetric

Metrics details

  • Behavioural methods

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD’s cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 print issues and online access

251,40 € per year

only 20,95 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

an annotation method

Data availability

The social mice dataset (CalMS21) used in this study is available online ( https://doi.org/10.22002/D1.1991 ) 24 , 51 . The single-monkey dataset used in this study is available in our GitHub repository ( https://github.com/YttriLab/asoid_paper/demo_dataset ) thanks to the laboratories of J. Zimmermann and B. Hayden at the University of Minnesota. The human dataset is part of the public NTU-RGB+D Action Recognition Dataset made available by the ROSE Laboratory at the Nanyang Technological University, Singapore 38 .

Code availability

The app, further documentation and the open-source code written in Python can be found at https://github.com/YttriLab/A-SOID (ref. 52 ) and is licensed under a modified BSD-3 license that permits unrestricted usage for non-commercial applications. The code to generate these figures is open-source and available in a GitHub repository at https://github.com/YttriLab/asoid_paper (ref. 53 ).

Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21 , 1281–1289 (2018).

Article   CAS   PubMed   Google Scholar  

Lauer, J. et al. Multi-animal pose estimation, identification and tracking with DeepLabCut. Nat. Methods 19 , 496–504 (2022).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Pereira, T. D. et al. SLEAP: a deep learning system for multi-animal pose tracking. Nat. Methods 19 , 486–495 (2022).

Graving, J. M. et al. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 8 , e47994 (2019).

Segalin, C. et al. The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice. eLife 10 , e63720 (2021).

Bala, P. C. et al. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nat. Commun. 11 , 1–12 (2020).

Article   ADS   Google Scholar  

Ro, S. et al. Simple Behavioral Analysis (SimBA) - an open source toolkit for computer classification of complex social behaviors in experimental animals. Preprint at bioRxiv https://doi.org/10.1101/2020.04.19.049452 (2020).

Hsu, A. I. & Yttri, E. A. B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nat. Commun. 12 , 1–13 (2021).

Luxem, K. et al. Identifying behavioral structure from deep variational embeddings of animal motion. Commun. Biol. 5 , 1–15 (2022).

Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88 , 1121–1135 (2015).

Schweihoff, J. F. et al. DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection. Commun. Biol. 4 , 1–11 (2021).

Article   Google Scholar  

Kane, G. A., Lopes, G., Saunders, J. L., Mathis, A. & Mathis, M. W. Real-time, low-latency closed-loop feedback using markerless posture tracking. eLife 9 , 1–29 (2020).

Nourizonoz, A. et al. EthoLoop: automated closed-loop neuroethology in naturalistic environments. Nat. Methods 17 , 1052–1059 (2020).

Klibaite, U. et al. Deep phenotyping reveals movement phenotypes in mouse neurodevelopmental models. Mol. Autism 13 , 1–18 (2022).

Giancardo, L. et al. Automatic visual tracking and social behaviour analysis with multiple mice. PLoS ONE 8 , e74557 (2013).

Article   CAS   PubMed   PubMed Central   ADS   Google Scholar  

De Chaumont, F. et al. Computerized video analysis of social interactions in mice. Nat. Methods 9 , 410–417 (2012).

Article   PubMed   Google Scholar  

Hong, W. et al. Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. Proc. Natl Acad. Sci. USA 112 , E5351–E5360 (2015).

Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S. & Branson, K. JAABA: interactive machine learning for automatic annotation of animal behavior. Nat. Methods 10 , 64–67 (2013).

von Ziegler, L., Sturman, O. & Bohacek, J. Big behavior: challenges and opportunities in a new era of deep behavior profiling. Neuropsychopharmacology 46 , 33–44 (2020).

Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface 11 , 20140672 (2014).

Article   PubMed   PubMed Central   Google Scholar  

Marshall, J. D. et al. Continuous whole-body 3D kinematic recordings across the rodent behavioral repertoire. Neuron 109 , 420–437 (2021).

Goodwin, N. L., Nilsson, S. R., Choong, J. J. & Golden, S. A. Toward the explainability, transparency, and universality of machine learning for behavioral classification in neuroscience. Curr. Opin. Neurobiol. 73 , 102544 (2022).

Berman, G. J. Measuring behavior across scales. BMC Biol. 16 , 1–11 (2018).

Caltech, J. J. S. et al. The multi-agent behavior dataset: mouse dyadic social interactions. Preprint at arXiv https://doi.org/10.48550/arXiv.2104.02710 (2021).

Karashchuk, P. et al. Anipose: a toolkit for robust markerless 3D pose estimation. Cell Rep. 36 , 109730 (2021).

Dunn, T. W. et al. Geometric deep learning enables 3D kinematic profiling across species and environments. Nat. Methods 18 , 564–573 (2021).

Günel, S. et al. Deepfly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila. eLife 8 , e48571 (2019).

Todd, J. G., Kain, J. S. & de Bivort, B. L. Systematic exploration of unsupervised methods for mapping behavior. Phys. Biol. 14 , 015002 (2017).

Article   PubMed   ADS   Google Scholar  

Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. GitHub https://github.com/slundberg/shap (2017).

Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2 , 56–67 (2020).

Mathis, M. W. & Mathis, A. Deep learning tools for the measurement of animal behavior in neuroscience. Curr. Opin. Neurobiol. 60 , 1–11 (2020).

Bohnslav, J. P. et al. DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels. eLife 10 , e63377 (2021).

Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16 , 321–357 (2002).

Blagus, R. & Lusa, L. SMOTE for high-dimensional class-imbalanced data. BMC Bioinform. 14 , 1–16 (2013).

Google Scholar  

Maldonado, S., López, J. & Vairetti, C. An alternative SMOTE oversampling strategy for high-dimensional datasets. Appl. Soft Comput. 76 , 380–389 (2019).

Winslow, J. T. Mouse social recognition and preference. Curr.Protoc. Neurosci. 22 , 1–8 (2003).

Yang, M., Loureiro, D., Kalikhman, D. & Crawley, J. N. Male mice emit distinct ultrasonic vocalizations when the female leaves the social interaction arena. Front. Behav. Neurosci. 7 , 159 (2013).

Shahroudy, A., Liu, J., Ng, T. T. & Wang, G. NTU RGB+D: a large scale dataset for 3D human activity analysis. Preprint at arXiv https://doi.org/10.48550/arXiv.1604.02808 (2016).

Sturman, O. et al. Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychopharmacology 45 , 1942–1952 (2020).

Datta, S. R., Anderson, D. J., Branson, K., Perona, P. & Leifer, A. Computational neuroethology: a call to action. Neuron 104 , 11–24 (2019).

McInnes, L., Healy, J. & Melville, J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. Preprint at arXiv https://doi.org/10.48550/arXiv.1802.03426 (2020).

Becht, E. et al. Dimensionality reduction for visualizing single-cell data using UMAP. Nat. Biotechnol. 37 , 38–44 (2018).

Packer, J. S. et al. A lineage-resolved molecular atlas of C. elegans embryogenesis at single-cell resolution. Science 365 , eaax1971 (2019).

van Unen, V. et al. Mass cytometry of the human mucosal immune system identifies tissue- and disease-associated immune subsets. Immunity 44 , 1227–1239 (2016).

Campello, R. J., Moulavi, D. & Sander, J. Density-based clustering based on hierarchical density estimates. Pacific-Asia Conference on Knowledge Discovery and Data Mining 10.1007/978-3-642-37456-2_14 (2013).

Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364 , eaav7893 (2019).

Article   CAS   Google Scholar  

Harris, C. R. et al. Array programming with NumPy. Nature 585 , 357–362 (2020).

Maisson, D. J.-N. et al. Widespread coding of navigational variables in prefrontal cortex. Curr. Biol. 33 , 3478–3488 (2023).

Voloh, B. et al. Hierarchical action encoding in prefrontal cortex of freely moving macaques. Cell Rep. 42 , 113091 (2023).

Friard, O. & Gamba, M. BORIS: a free, versatile open-source event-logging software for video/audio coding and live observations. Methods Ecol. Evol. 7 , 1325–1330 (2016).

Sun, J. J., Kennedy, A., Zhan, E., Yue, Y. & Perona, P. Task programming: learning data efficient behavior representations. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. https://doi.org/10.1109/CVPR46437.2021.00290 (2021).

Tillmann, J. F., Hsu, A. I., Schwarz, M. K. & Yttri, E. A. A-soid: an active learning platform for expert-guided, data efficient discovery of behavior. Zenodo https://doi.org/10.5281/zenodo.10210509 (2023).

Tillmann, J. F., Hsu, A. I., Schwarz, M. K. & Yttri, E. A. Paper code repository for an active learning platform for expert-guided, data efficient discovery of behavior. Zenodo https://doi.org/10.5281/zenodo.10257993 (2023).

Download references

Acknowledgements

We thank staff at the laboratories of J. Zimmermann and B. Hayden at the University of Minnesota for their patience and unpublished 3D OpenMonkeyStudio pose data. We also thank A. Kennedy, J. Sun, D. Anderson and the team at the California Institute of Technology and Northwestern University, USA, who created the CalMS21 dataset to provide a comprehensive dataset that can be used to benchmark current and future approaches to the classification of social behavior in mice. Portions of the research in this paper used the NTU-RGB+D Action Recognition Dataset made available by the ROSE Laboratory at the Nanyang Technological University, Singapore.

This work was supported by German Research Foundation (SFB1089 P02, P03 and B06 to M.K.S.; SPP 2041 SCHW1578/2-1 to M.K.S.). Research in the Schwarz laboratory was also supported by the Verein zur Förderung der Epilepsieforschung e.V. (M.K.S. and J.F.T.) and from the program Netzwerke 2021, an initiative of the Ministry of Culture and Science of the State of Northrhine Westphalia (M.K.S.). Research in the Yttri laboratory was supported by the Brain Research Foundation, the Kaufman Foundation and the US-Israel Binational Science Foundation (E.A.Y. and A.I.H.).

Author information

These authors contributed equally: Jens F. Tillmann, Alexander I. Hsu.

Authors and Affiliations

Institute of Experimental Epileptology and Cognition Research, University of Bonn, Bonn, Germany

Jens F. Tillmann & Martin K. Schwarz

Department of Biological Sciences, Carnegie Mellon University, Pittsburgh, PA, USA

Alexander I. Hsu & Eric A. Yttri

Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, USA

Eric A. Yttri

You can also search for this author in PubMed   Google Scholar

Contributions

J.F.T. and A.I.H. contributed equally to this paper. M.K.S. and E.A.Y. contributed equally and are both corresponding authors. J.F.T., A.I.H., M.K.S. and E.A.Y. wrote and reviewed the paper. J.F.T. proposed the idea and A.I.H. designed the core functionality and main analysis scripts. Both J.F.T. and A.I.H. created the app and worked on analysis and figures. J.F.T. annotated the primate data. E.A.Y. and M.K.S. provided support and funds.

Corresponding authors

Correspondence to Martin K. Schwarz or Eric A. Yttri .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Methods thanks anonymous reviewers for their contribution to the peer review of this work. Primary Handling Editor: Nina Vogt, in collaboration with the Nature Methods team. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1 active-learning framework on user-defined data..

a) The A-SOiD GUI offers step-by-step navigation to run A-SOiD on a user’s own data. b) First users can select the origin/type of their pose estimation data (SLEAP, DLC, or CALMS21) and uploads their dataset including a previously labeled ground truth. Right, the user can enter basic parameters (frame-rate, resolution), behavioral categories of interest that are contained in the ground truth dataset as well as sub-select individuals/animals and key points/body parts as a basis for feature extraction. c) After input of a temporal reference frame (aka. bout length) for feature extraction using a histogram as shown in Fig. 1 (top), features are extracted, and a number of splits are provided to evaluate later classification training. d) In the active-learning segment, a classifier is trained by the iterative addition of low-confidence predictions. Here, refinement is directly taken from the remaining ground truth. During each iteration, the model’s performance is evaluated on a held-out test data for multiple splits. This process can be viewed live for each iteration. e) Finally, once the training is complete, users can use the app to upload new unlabeled data and use the previously trained model for classification. f) After classification, the app allows users to go through the results and view a brief report. g) Users are also able to discover conserved patterns in their ground truth data by selectively clustering annotation classes with directed unsupervised classification. Subtypes of interest can then be exported to create a new training set and be used to train a classifier.

Extended Data Fig. 2 Extended embeddings and feature windows.

a) Different embeddings of the features extracted from the CalMS21 dataset (feature window = 400 ms or 12 frames). Left, UMAP embedding as seen in Fig. 1g . Middle, principal component analysis (PCA, n = 2). Right, t-Distributed Stochastic Neighbor Embedding (t-SNE, n = 2). The annotations for each underlying behavior are superimposed onto the embedding (attack = red, investigation = orange, mount = blue, other = dark gray). Inserts show the underlying distribution separated by behavior (light gray). b) 2D UMAP embedding of CalMS21 features provided with the dataset. The features include the pose estimation of all body parts and 32 additional features extracted with TREBA[53]. c) 2D UMAP embeddings of feature bins across a range of 2 frames to 150 frames. Note that 2 frames (top left) is the minimum with a 30 Hz frame-rate, and 150 frames (bottom right) is considered very coarse for most observations. d) Adjusted mutual information score (AMI, black line) between the assignments and the original human annotations for a minimum cluster size range of 0.5 to 10.0% with 0.5% intervals. An AMI score of 1.0 indicates a perfect overlap even if the number of groups is different- that is, if investigation was perfectly represented by two clusters instead of one. In addition, we investigated the total number of clusters (gray, dashed line) as we expected a higher split to be more likely to incorporate the subtle differences between the behaviors. e) Selected HDBscan clustering of the embedded features (400 ms) in d. Left, the highest AMI reached a minimum cluster size of 6.0% (AMI = 0.267). Right, the highest number of clusters was found with a minimum cluster size of 1.5% (AMI = 0.195). Colors show identified clusters within each plot. Note that samples that cannot be confidently associated with a cluster are collectively annotated as noise (dark gray) and can therefore span the entire embedding. f) 2D histogram of the assigned cluster groups (y-axis) in d in relation to their ground truth annotations. Each histogram is normalized so that the sum of each column (ground truth behavior, for example, attack) is 1.0.

Extended Data Fig. 3 More candidates for active-learning refinement appear at behavior transitions.

a) Transition matrix for the frame annotation that happens before refinement candidates throughout A-SOiD (left), when compared to a random frame selection of these behaviors (right). b) Adjusted mutual information score as a metric to quantify similarity between prior frame (t-1) and refinement candidate/random selection (t). n=20 random seeds for sampling. Box spans first to third quartile, with the whiskers extending below first and beyond third quartiles by 1.5 times of inter-quartile range. One-sided t-test, p=8.887e-32. c) Transition matrix for the frame annotation that happens after refinement candidates throughout A-SOiD (left), in contrast to a random frame selection of these behaviors (right). d) Adjusted mutual information score as a metric to quantify similarity between next frame (t+1) and refinement candidate/random selection. n=20 random seeds for sampling. Box spans first to third quartile, with the whiskers extending below first and beyond third quartiles by 1.5 times of inter-quartile range. One-sided t-test, p=1.154e-34.

Extended Data Fig. 4 SHAP-replicated decision paths of two instances where A-SOiD, but not using full training data at once, correctly predicted “attack” and “investigation”.

a) Multi-output decision plot on an instance where human-annotated “attack”, but A-SOiD iteration 0 predicted “investigation”. The features on the x-axis are ranked sorted by their impact on cumulative prediction probability on the y-axis. The highest final prediction probability of the four color-coded lines revealed the prediction was “investigation”. b) Multi-output decision plot on an instance where human-annotated “investigation”, but A-SOiD iteration 0 predicted “other”. The features on the x-axis are ranked and sorted by their impact on cumulative prediction probability on the y-axis. The highest final prediction probability of the four color-coded lines revealed the prediction was “other”. c) Similar to a), a multi-output decision plot on an instance where human-annotated “attack”, but full training data at once incorrectly predicted “investigation”. d) Similar to b), a multi-output decision plot on an instance where human-annotated “investigation”, but full training at once incorrectly predicted “other”. e) Similar to a) and c), a multi-output decision plot on an instance where human-annotated “attack”, and A-SOiD iteration 20 correctly predicted “attack”. f) Similar to b) and d), a multi-output decision plot on an instance where human-annotated “investigation”, and A-SOiD iteration 20 correctly predicted “investigation”. Red: “attack”; Orange: “investigation”; Blue: “mount”; Black: “other”. Features names are black when they’re inter-distance, gray when they’re speed, and teal when they’re angular change. R: resident; I: intruder.

Extended Data Fig. 5 Benchmark results against state-of-the-art supervised classification methods.

a) Predicted ethogram (including “other”) using SimBA (blue), DeepEthogram (DEG, purple), and A-SOiD (pink) against target human annotation (black) without “other”. Every fiftieth frame is shown to allow for better visualization. Session n=19. b) Micro F1 scores across all behaviors (including classification of “other”) using SimBA (blue), DEG (purple), and A-SOiD (pink). Error bars represent +/- 3 SD across 20 seeds. c) Weighted macro F1 scores averaged across all behaviors (including classification of “other”) using SimBA (blue), DEG (purple), and A-SOiD (pink). Error bars represent +/- 3 SD across 20 seeds. d) The percentages of each behavior predicted for each algorithm (top left: SimBA; DEG: top right; A-SOiD: bottom left; human annotation: bottom right). e) The total number of frames being predicted as “attack”, “investigation”, “mount”, and “other”, for each algorithm (top left: SimBA; DEG: top right; A-SOiD: bottom left; human annotation: bottom right). f) F1 scores for individual behavior using SimBA (blue), DEG (purple), and A-SOiD (pink). Error bars represent +/- 3 SD across 20 seeds. g) F1 scores for individual behavior using unbalanced (unbal, black), random under-sampling (under, brown), SMOTE (maroon), and A-SOiD (pink). Error bars represent +/- 3 SD across 20 seeds.

Extended Data Fig. 6 Active learning speed across different sample sizes.

To estimate the total time it takes to run larger datasets with A-SOiD, we timed the time it takes for our active-learning regime (max iterations = 20, max number of samples per iteration = 200, initial ratio = 0.01) across a range of subsets (0.3 to 1.0) of the CalMS21 dataset (number of features = 100). We then fit a linear function to the measurements to estimate the performance speed with increasing sample sizes. a) Total time A-SOiD takes to complete 20 iterations, including feature extraction of the train set. Given the fit, every 1000 new samples increase the runtime by 3 seconds. The time it takes to run 1 Million samples is roughly 53 min. b) Isolated feature extraction speed for each subset. Given the fit, every 1000 samples increase the runtime by 2 seconds (1M samples about 28 min). Notably, we considerably optimized feature extraction by employing just-in-time compilation using the Python implementations of numba 0.52.0 ( https://github.com/numba/numba ). However, the feature extraction, which is run once in the beginning, is still the major bottleneck when it comes to speed. The vertical dotted line indicates the original size of the dataset. The vertical dotted line indicates the original size of the dataset. Each subset was repeated 3 times, and the speed was averaged across seeds. Error bars represent the standard deviation.

Extended Data Fig. 7 SHAP-replicated decision paths of two instances where A-SOiD, but not using full training data at once, correctly predicted “jump” and “climb (S)”.

a) Multi-output decision plot on an instance where human-annotated “jump”, but A-SOiD iteration 0 predicted “walk”. The features on the x-axis are ranked and sorted by their impact on cumulative prediction probability on the y-axis. The highest final prediction probability of the four color-coded lines revealed the prediction was “walk”. b) Multi-output decision plot on an instance where human-annotated “climb (S)”, but A-SOiD iteration 0 predicted “rear”. The features on the x-axis are ranked and sorted by their impact on cumulative prediction probability on the y-axis. The highest final prediction probability of the four color-coded lines revealed the prediction was “rear”. c) Similar to a), a multi-output decision plot on an instance where human-annotated “jump”, but full training data at once incorrectly predicted “walk”. d) Similar to b), a multi-output decision plot on an instance where human-annotated “climb (S)”, but full training at once incorrectly predicted “rear”. e) Similar to a) and c), a multi-output decision plot on an instance where human-annotated “jump”, and A-SOiD iteration 20 correctly predicted “jump”. f) Similar to b) and d), a multi-output decision plot on an instance where human-annotated “climb (S)”, and A-SOiD iteration 20 correctly predicted “climb (S)”. Orange: “ceiling climb” (Climb C); Yellow: “sidewall climb” (Climb S); Pink: “jump”; Green: “rear”; Blue: “walk”. Features names are black when they’re inter-distance, gray when they’re speed, and teal when they’re angular change. R: right-sided body parts; L: left-sided body parts.

Extended Data Fig. 8 A-SOiD’s performance on data that contains a higher number of actions, including more complex actions (NTU-RGB-D60 dataset; see Methods).

Social human actions (a-e) and Single human actions (g-h). a) Schematic representation of pose estimation skeletons in NTU-RGB-D60 dataset. b) A-SOiD performance (F1 score, 3 cross-validations, blue) on held-out test data of the last iteration (50 iterations; n_samples = 81045, 77% of training set) plotted versus a classifier trained with the same number of samples (partial, red) and the full training set (full, black). See Supplementary Table 6 for details. c) Performance (F1 score, 3 cross-validations) on held-out test data is plotted against active-learning iterations. Black: Unweighted class average. The dashed line demonstrates unweighted class average performance using full training annotations at once. Data presented as mean values +/- standard deviation across 3 random initializations. d) Stacked bar graph of the number of training annotations is plotted against active-learning iterations (right-most: full annotation count). Bars represent average training samples across 3 seeds. Dashed line indicates the number of samples in the final iteration (77%, n = 81045). e) Confusion matrices where prediction mistakes were being made for the last iteration (A-SOiD, top) and a classifier trained on the same number of samples, randomly sampled from the training set (Partial, bottom). Darker red shades along the diagonal indicate better algorithm performance in matching ground truth. f) Performance across all 49 actions (F1 score, 3 cross-validations, blue) on held-out test data of the last iteration (90 iterations; n_samples = 546245, 87% of training set) plotted versus the performance of a classifier trained with the same number of samples (partial, red) and with full training set (full, black). Data presented as mean values +/- standard deviation across 3 random initializations. See Supplementary Table 7 for details. g) Performance (F1 score, 3 seed cross-validations) on held-out test data is plotted against active-learning iterations. Ten representative actions, including the worst and best performing, are shown. Dashed lines demonstrate performance using full training annotations at once. Black: Unweighted class average across all 49 actions. Error bars represent the standard deviation across 3 initializations. h) Same as e but for the 49 actions subset.

Extended Data Fig. 9 A-SOiD performance done on 3, 5, or 7 key points.

a) Identical to what was used in the Fig. 2 , 5 key points were used (nose, neck, two hips, and tail base). Data presented as mean values +/- standard deviation across 20 random initializations. b) Average A-SOiD performance on using just 3 key points (nose, neck, and tail base). Data presented as mean values +/- standard deviation across 20 random initializations. c) Average A-SOiD performance on using all 7 key points (nose, two ears, neck, two hips, and tail base). Data presented as mean values +/- standard deviation across 20 random initializations. d) Average F1 score difference between 5 key points and either all 7 key points or a reduced 3 key points over active-learning iterations.

Supplementary information

Supplementary information.

Supplementary Discussion, Supplementary Fig. 1, Supplementary Note and Supplementary Tables 1–7.

Reporting Summary

Peer review file, supplementary video 1.

Video examples of two subclasses segmented from investigation that reflect anogenital investigation. On the left, one mouse directly approaches the anogenital area of another mouse, irrespective of the incoming angle (anogenital approach). On the right, one mouse investigates the anogenital area of another mouse while already being in close proximity to begin with (anogenital investigation).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Tillmann, J.F., Hsu, A.I., Schwarz, M.K. et al. A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior. Nat Methods (2024). https://doi.org/10.1038/s41592-024-02200-1

Download citation

Received : 04 November 2022

Accepted : 29 January 2024

Published : 21 February 2024

DOI : https://doi.org/10.1038/s41592-024-02200-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

an annotation method

  • Search Menu
  • Author Guidelines
  • Submission Site
  • Open Access
  • About Briefings in Bioinformatics
  • Journals Career Network
  • Editorial Board
  • Advertising and Corporate Services
  • Self-Archiving Policy
  • Journals on Oxford Academic
  • Books on Oxford Academic

Issue Cover

Article Contents

Introduction, performance evaluation, results comparison, limitations and future implications, author biographies.

  • < Previous

scEVOLVE: cell-type incremental annotation without forgetting for single-cell RNA-seq data

  • Article contents
  • Figures & tables
  • Supplementary Data

Yuyao Zhai, Liang Chen, Minghua Deng, scEVOLVE: cell-type incremental annotation without forgetting for single-cell RNA-seq data, Briefings in Bioinformatics , Volume 25, Issue 2, March 2024, bbae039, https://doi.org/10.1093/bib/bbae039

  • Permissions Icon Permissions

The evolution in single-cell RNA sequencing (scRNA-seq) technology has opened a new avenue for researchers to inspect cellular heterogeneity with single-cell precision. One crucial aspect of this technology is cell-type annotation, which is fundamental for any subsequent analysis in single-cell data mining. Recently, the scientific community has seen a surge in the development of automatic annotation methods aimed at this task. However, these methods generally operate at a steady-state total cell-type capacity, significantly restricting the cell annotation systems'capacity for continuous knowledge acquisition. Furthermore, creating a unified scRNA-seq annotation system remains challenged by the need to progressively expand its understanding of ever-increasing cell-type concepts derived from a continuous data stream. In response to these challenges, this paper presents a novel and challenging setting for annotation, namely cell-type incremental annotation. This concept is designed to perpetually enhance cell-type knowledge, gleaned from continuously incoming data. This task encounters difficulty with data stream samples that can only be observed once, leading to catastrophic forgetting. To address this problem, we introduce our breakthrough methodology termed scEVOLVE, an incremental annotation method. This innovative approach is built upon the methodology of contrastive sample replay combined with the fundamental principle of partition confidence maximization. Specifically, we initially retain and replay sections of the old data in each subsequent training phase, then establish a unique prototypical learning objective to mitigate the cell-type imbalance problem, as an alternative to using cross-entropy. To effectively emulate a model that trains concurrently with complete data, we introduce a cell-type decorrelation strategy that efficiently scatters feature representations of each cell type uniformly. We constructed the scEVOLVE framework with simplicity and ease of integration into most deep softmax-based single-cell annotation methods. Thorough experiments conducted on a range of meticulously constructed benchmarks consistently prove that our methodology can incrementally learn numerous cell types over an extended period, outperforming other strategies that fail quickly. As far as our knowledge extends, this is the first attempt to propose and formulate an end-to-end algorithm framework to address this new, practical task. Additionally, scEVOLVE, coded in Python using the Pytorch machine-learning library, is freely accessible at https://github.com/aimeeyaoyao/scEVOLVE .

Single-cell RNA sequencing (scRNA-seq) technologies enable the profiling of gene expressions in millions of individual cells, offering unparalleled insights into the intricate cellular ecosystem [ 1 , 2 ]. Crucial to analyzing scRNA-seq data is cell annotation since numerous downstream explorations hinge on the identification of specific cell types [ 3 , 4 ]. A conventional single-cell annotation pipeline typically commences with the segmentation of cell data into various clusters, proceeds to identify each group's marker genes by differential expression analysis and culminates with group annotation through gene ontology knowledge. For instance, scCatch assigns annotation categories to cells based on canonical cell-type marker genes through a scoring approach grounded on evidence [ 5 ]. Nonetheless, this methodology necessitates manual verification of marker genes, entailing the review of extensive biological literature [ 6 , 7 ]. Consequently, this process could pose a considerable challenge to researchers within nonspecialized fields [ 8 ].

As the availability of extensive, well-labelled reference data expands, numerous automated annotation strategies, based on projection and classification, have been emerging [ 9 , 10 ]. For example, scmap utilizes new cells, projects them onto reference data and measures their similarity to known cluster centroids within the reference data to carry out annotation [ 11 ]. Seurat, conversely, predicts the supervised targets on the data by identifying pairs of anchor cells and subsequently building shared neighbor networks between the reference and target data [ 12 ]. Capitalizing on the advanced capabilities of deep learning in feature representation, studies have begun to employ nonlinear neural networks to facilitate cell classification. A case in point is scANVI, a generative model founded on a deep variational autoencoder-based scRNA-seq semi-supervised framework, developed with the aim of leveraging any available cell state annotations [ 13 ]. MARS adopts meta-learning in such a manner that identical cell types will share similar features, while any discrepancies in cell type will result in quite distinct features [ 14 ]. On a similar note, scArches employs domain adaptation and parameter optimization techniques to construct references and contextualize target data [ 15 ]. In conclusion, the majority of current annotation methods operate within the boundaries of established cell-type knowledge capacity, which, unfortunately, restricts their ability to discover new cell types from an influx of data. This limitation poses significant challenges to their applicability in real-world situations.

The emerging prominence of incremental learning has drawn scholarly interest in models capable of an ongoing learning process. This involves accommodating new tasks through the assimilation of fresh data while retaining previously acquired knowledge [ 16 , 17 ]. A significant challenge posed by incremental learning involves averting catastrophic forgetting, a scenario where the performance on former tasks or domains suffers a significant decline over time upon the introduction of new ones [ 18 , 19 ]. This challenge emanates from a broader issue in neural networks referred to as the stability-plasticity dilemma. Here, plasticity correlates with the capacity to incorporate new knowledge, while stability means retaining prior knowledge during encoding. Incremental learning research largely falls into three categories, differentiated by how task-specific information is stored and applied in the sequential learning process. These categories include replay methods [ 20 , 21 ], regularization-based techniques [ 22 , 23 ] and parameter isolation methods [ 24 , 25 ]. For an in-depth overview of related studies, refer to the existing literature [ 26 , 27 ].

Drawing upon incremental learning theories, it is rational to infer that the scRNA-seq annotation system operates on an inherently incremental basis, where it consistently integrates new cell heterogeneity information while retaining existing knowledge. To illustrate, researchers examining pancreatic tissue can discover numerous new cell types without losing their understanding of the cell types in intestinal and stomach tissues. Contrarily, the majority of annotation systems operate within a batch setting, implying they require prior knowledge of all cell types and simultaneous accessibility to all types'training data [ 28–30 ]. Notwithstanding, the sphere of scRNA-seq data analysis is progressively converging with artificial intelligence, demanding more adaptable approaches to managing the vast-scale and dynamic nature of practical cell annotation scenarios. Fundamentally, a solitary cell annotation system should possess the capability to progressively learn about novel cell types predicated on the availability of their training data. This system modality is referred to as incremental cell-type annotation.

To be classified as an incremental cell-type annotation algorithm, it should meet several formal criteria. Firstly, it should be capable of being trained from a continuous data stream where cells from varying cell types appear at different intervals. Secondly, it should deliver a competitive multi-class classifier for the cell types identified up to that point. Lastly, it should adhere to a stringent set of computational specifications and memory usage, or at least demonstrate a marginal increase in proportion to the number of observed cell types thus far. The initial two criteria underscore the practical applicability of these incremental annotation tasks in classifying cell types. The final specification rejects any simple algorithms that might, for instance, retain all training samples and employ a standard multi-class classifier whenever a new dataset is introduced [ 14 , 31 ].

Despite significant advancements in automatic scRNA-seq annotation methods over recent years, a fully satisfactory cell-type incremental annotation algorithm remains elusive. Most current methodologies fall short of requirements 1 or 2, as they solely manage a predetermined number of cell types or necessitate simultaneous availability of all training data. Training classifiers from incrementally obtained cell-type data streams, such as through the application of stochastic gradient descent optimization, offers a potential solution. Yet, this approach often leads to a rapid diminution in annotation accuracy, a phenomenon we refer to as catastrophic forgetting in incremental learning. Notwithstanding the few techniques currently meeting the aforementioned criteria, their utility remains largely confined to scenarios with fixed data representation in the single-cell domain. Consequently, these techniques cannot be expanded to incremental architectures, which simultaneously learn continual classifiers and features, thus leaving them outdated in relation to annotation accuracy.

This paper introduces scEVOLVE, a new framework for incremental annotation of cell types that can seamlessly integrate new data without disregarding established knowledge. The framework advances from the standpoint of contrastive sample replay and cell-type decorrelation. To offset the risk of erasing prior cell-type information with the introduction of new data, scEVOLVE employs an episodic memory buffer. This feature stores a sampling of prior stage data to then be used in conjunction with the current stage. The selection of sample examples applies a protocol rooted in nearest prototype classification certainty; this protocol allows for an effective reduction of duplicate samples throughout the runtime while ensuring high proximity to the cell-type prototype. In the next stages, the framework could replay previous samples. However, conventional softmax-based prototype classification may contribute to bias issues resulting from cell-type imbalance. This can lead to the misplacement of many existing cell-type samples into new categories. To counter this, scEVOLVE employs a contrastive replay method. No longer relying on sample-to-anchor pairs, this method utilizes prototype-to-anchor pairs in the contrastive-based loss, strictly using prototypes that align with the cell types present in the same batch. The incremental model of scEVOLVE is carefully designed to mirror the oracle model in every phase. The goal is to make the full data-trained performance of the oracle model the ultimate target. To get there, we impose uniform scattering across cell-type data representations in each phase, emulating the oracle model representations. Partition confidence maximization drives the cell-type decorrelation regularization, making this model implementation specific.

To judge scEVOLVE's efficacy, we've selected several large-scale scRNA-seq datasets and designed an exhaustive set of cell-type incremental annotation benchmarks. A multitude of experiments using these benchmarks reveals scEVOLVE's power to mitigate the catastrophic forgetting problem while allowing incremental cell-type learning over extended timeframes. Most importantly, scEVOLVE manages the forgetting problem by balancing gradient propagation and regularizing the scatter of embedding representations. Further investigation verifies the effectiveness of every module. Lastly, we believe our method can be comfortably incorporated into a variety of deep softmax-based, single-cell annotation techniques, making it a worthy candidate for widespread integration.

We commence our discussion with the establishment of the problem context and the associated notations. In order to closely align with the practical scenario, we understand that scRNA-seq data are stored in a data flow and subsequently analyzed by the model in a cell type-incremental format. Specific sample sets illustrated by |$\{\mathcal{X}^{1},\mathcal{X}^{2},...\}$| and corresponding label sets denoted by |$\{\mathcal{Y}^{1},\mathcal{Y}^{2},...\}$| could either originate from the same or disparate scRNA-seq datasets. To enhance the efficacy of model learning, we segregate the data stream into a series of learning tasks, represented as |$\mathcal{D}=\{ \mathcal{D}^{i} \}_{i=1}$|⁠ . Here, |$\mathcal{D}^{i}=\{\mathcal{X}^{i} \times \mathcal{Y}^{i}, \mathcal{C}^{i}\}$| includes sample sets |$\mathcal{X}^{i}$|⁠ , the corresponding label sets |$\mathcal{Y}^{i}$|⁠ , as well as task-specific cell types |$\mathcal{C}^{i}$|⁠ . Each sample set, |$\mathcal{X}^{i}$|⁠ , is hypothetically divided at random into a training dataset and a testing dataset. These can be denoted, respectively, as |$\mathcal{X}^{ir}=\{(x_{j}^{ir},y_{j}^{ir})_{j=1}^{n_{ir}}\}$| and |$\mathcal{X}^{it}=\{(x_{j}^{it},y_{j}^{it})_{j=1}^{n_{it}}\}$|⁠ . Within these, the labels within the training dataset, |$y_{j}^{ir}$|⁠ , are known; however, the labels within the test set, |$y_{j}^{it}$|⁠ , are yet to be predicted. Furthermore, the label relationship among any two given datasets within the data flow may exhibit partial overlap. This is represented by the conditions |$\mathcal{C}^{i} \cap \mathcal{C}^{j} \neq \varnothing $|⁠ , |$\mathcal{C}^{i} \setminus \mathcal{C}^{\,j} \neq \varnothing $|⁠ , and |$\mathcal{C}^{\,j} \setminus \mathcal{C}^{i} \neq \varnothing $|⁠ .

While many annotation methods have performed satisfactorily under constant total cell-type scenarios in recent years, the introduction of a new, unrelated dataset can trigger what is often referred to as catastrophic forgetting issues. This phenomenon is characterized by the erasure of previously learned information about old cell types. Yet, our natural expectation is a continual evolution of the annotation system over time, accommodating an increasing repertoire of cell types. As such, our aim is to formulate a universal algorithm that supports incremental learning of new cell types, reflecting the ever-changing demands of real-world cell annotation. With this algorithm, when cells from new types need annotating, the model will adapt, recognizing their gene expression patterns without neglecting the knowledge previously acquired. Repeated training of all cells from pre-existing cell types will be eliminated, resulting in a significant rise in the efficiency of the cell annotation system. Moreover, our task is executed within an inductive learning framework, a divergence from the transductive learning approach adopted by most prior domain adaptation-based single-cell annotation methods [ 32–35 ]. Such a setup considers the practical implications of deploying the cell annotation system in real-world environments without incurring additional costs associated with adjusting the test data.

Basic network construction of model

Our model is comprised of a Zero-Inflated Negative Binomial-oriented denoising autoencoder [ 36 ], possessing a reconstruction loss denoted as |$L_{zinb}$|⁠ , and a prototype-parameterized classifier symbolized as |$\Phi $| (refer to Figure 1 ). Comprehensive details can be found in the Supplementary Information (SI) . The classifier, |$\Phi $|⁠ , which is appended subsequent to the encoder labeled |$z=f_{e}(x)$|⁠ , can be delineated as |$\Phi =\langle z, V \rangle /\tau $|⁠ . Herein, |$V=[v_{1},...]^{T}$| houses the trainable cell-type prototypes learned thus far, |$\langle \cdot ,\cdot \rangle $| signifies the cosine similarity, and |$\tau $| is a scaling factor. Subsequently, the similarity vector is normalized by the softmax function to derive the probability of a sample’s alignment with different prototypes. As an example, consider the |$s$| -stage of training, where all learned cell types are abbreviated as |$\mathcal{C}_{1:s}^{r}= \bigcup _{i=1}^{s}\mathcal{C}^{ir}$|⁠ . The likelihood that sample |$x_{i}$| pertains to the cell type |$c$| is represented in

The following text provides an overview of scEVOLVE: (A) During the $s$th stage, the model updates the exemplar sets based on the dataset from the $s-1$th stage. This updated set, in conjunction with the $s$th stage dataset, forms the model’s input. When new cell types emerge at the $s$th stage, matching new heads are incorporated into the classifier. The formulation $\mathcal{L}_{pbc}^{s}$ is devised to balance the gradient progression between existing and novel cell types, while the formula $\mathcal{L}_{cwd}^{s}$ enforces a uniform scattering of data representations pertaining to different cell types in the embedding space. (B) The text includes an illustration depicting the procedure of gradient propagation from individual samples to all prototypes under various learning strategies. Our approach more effectively manages the gradient propagation process, thereby enhancing the identification of both new and existing cell types. (C) A further illustration demonstrates the compression technique applied to the global partition measurement values, excluding the diagonal elements, facilitating the learning process to achieve the most confident and promising classification solution.

The following text provides an overview of scEVOLVE: ( A ) During the |$s$| th stage, the model updates the exemplar sets based on the dataset from the |$s-1$| th stage. This updated set, in conjunction with the |$s$| th stage dataset, forms the model’s input. When new cell types emerge at the |$s$| th stage, matching new heads are incorporated into the classifier. The formulation |$\mathcal{L}_{pbc}^{s}$| is devised to balance the gradient progression between existing and novel cell types, while the formula |$\mathcal{L}_{cwd}^{s}$| enforces a uniform scattering of data representations pertaining to different cell types in the embedding space. ( B ) The text includes an illustration depicting the procedure of gradient propagation from individual samples to all prototypes under various learning strategies. Our approach more effectively manages the gradient propagation process, thereby enhancing the identification of both new and existing cell types. ( C ) A further illustration demonstrates the compression technique applied to the global partition measurement values, excluding the diagonal elements, facilitating the learning process to achieve the most confident and promising classification solution.

It is pertinent to note that as observed cell types increase, so does the number of classification heads. In the |$s$| th training phase, the model has access solely to the |$\mathcal{D}^{s}$| data related to that phase, and each sample can only be observed once. The classification objective utilizes the cross-entropy function, expressed in

From the provided formula, it is clear that |$L_{ind}^{s}$| trains the model solely using current stage data, a method frequently referred to as individual training. This approach does not aim to prevent forgetting. Contrastingly, joint learning represents the other extremity, where the model is trained utilizing all samples from both prior |$s-1$| stages, and the current |$s$| th stage. Joint learning’s cross-entropy loss is represented in

As such, the performance of joint learning can be posited as the upper boundary of our task.

Exemplar set construction and management

Firstly, this study investigates the underlying causes of catastrophic forgetting during cell-type incremental annotation, examining it through the lens of gradient propagation. The calculation of the gradient for a single data point or sample represented as |$x$| with its corresponding label |$y$| at stage |$s$| can be determined as depicted in

Here, the symbol |$c$| denotes a specific cell type within the existing label set. Drawing from this equation, it can be inferred that the dimension of the gradient vector corresponding to |$y$| experiences a decrease by |$p_{sy}-1<0$|⁠ , while the alternate dimension sees an increase by |$p_{sc}>0$|⁠ . Applying the chain rule in conjunction with this, the process of model updating will furnish a positive gradient for the prototype of the cell type represented as |$y$| with |$v_{y}^{s}=v_{y}^{s}-\eta f_{e}^{s}(x)(p_{sy}-1)$|⁠ , and disperse a negative gradient to the rest of the prototypes with |$v_{c}^{s}=v_{c}^{s}-\eta f_{e}^{s}(x)p_{sc}$|⁠ . This implies that upon adjusting the model by directly optimizing the loss function, depicted as |$L_{ind}^{s}$|⁠ , it can be inferred that the learning process associated with new cell types governs gradient propagation. This results in the occurrence of catastrophic forgetting.

Based on the above analysis, a critical step in managing gradient propagation and averting the forgetting problem is the storage of previously experienced samples. However, this necessitates considering two prerequisites for storing samples. Commonly referred to as joint learning, the method of storing all training samples previously encountered necessitates considerable memory resources, particularly as the quantum of previously learned samples mounts. Conversely, a haphazard approach of storing samples oblivious of their cell type may result in certain cell types left without any stored samples—thus deleteriously affecting model performance. Given these two prerequisites, careful selection of a prior sample subset as representative samples stored in the memory buffer |$\mathcal{M}$| to augment the current training set is essential. With stage |$s$| exemplifying, the observed cell types thus far can be represented as |$\{\mathcal{C}^{1},\mathcal{C}^{2},...,\mathcal{C}^{s-1}\}$|⁠ , while the quantity of cell types newly introduced at stage |$i$| can be represented as |$k_{i}=|\mathcal{C}^{i} \setminus (\bigcup _{k=1}^{i-1}\mathcal{C}^{k} \cap \mathcal{C}^{i})|$|⁠ . Subsequently, the representative sample sets at stage |$s$|⁠ , denoted |$E_{1}^{s}$|⁠ ,..., |$E_{k_{1:s-1}}^{s}$|⁠ , are dynamically contrived from the data stream with |$k_{1:s-1}=\sum _{i=1}^{s-1}k_{i}$|⁠ . The training set for the |$s$| th time period, denoted as |$\mathcal{X}^{s*}$|⁠ , can be articulated as |$\mathcal{X}^{s*}=\mathcal{X}^{s}\cup \bigcup _{i=1}^{k_{1:s-1}}E_{i}^{s}$|⁠ . To constrain memory requirements, hyperparameter |$m$| is deemed a fixed value, representing the quantity of representative samples for each cell type. During the revolutionary selection process, the selected representatives are modelled to closely mimic the corresponding cell-type prototype. For a specific cell type |$\mathcal{C}^{cr}$| at stage |$s$|⁠ , the similarity of each constituent sample |$x_{ic}^{sr}$| to its corresponding prototype |$v_{c}^{s}$| can be determined as |$s_{ic}=\langle f_{e}^{s}(x_{ic}^{sr}), v_{c}^{s} \rangle /\tau $|⁠ , allowing us to single out the top |$m$| samples with the highest similarity as representatives for that cell type. Furthermore, there’s no requirement to reselect a representative set for previously learned cell types at the current stage. Once the representative sample set has been assembled, it simplifies the broad implementation of the loss function |$L_{ind}^{s}$|⁠ , as depicted in

Prototypical contrastive replay learning

While we establish a corresponding exemplar set for each distinct cell type, the issue of an imbalanced gradient propagation persists during batch training with |$L_{rep}^{s}$|⁠ . The feature extractor specifically focuses its attention on the attributes of newly introduced cell types. This results in the positioning of new and old cell samples in close proximity within the embedding space, thereby facilitating the categorization of samples into new cell clusters. As training progresses, the gradient corresponding to old cell types proves inadequate, given that each cell type’s representative samples are fixed in the memory buffer. The newly introduced cell types then monopolize this process, rendering their respective samples highly distinguishable, while simultaneously causing those of the old cell types to become almost indiscernible. Therefore, efficiently governing the gradient flow between the old and new cell types could significantly favor the mitigation of the forgetting problem.

Based on the memory buffer |$\mathcal{M}$|⁠ , a selective retraining of previous samples of old cell types can be used to regulate gradient propagation. This regulation can be accomplished by choosing particular anchor-to-prototype pairs and computing their classification objective. This strategy may curb the imbalance of gradient propagation by guiding samples back to the corresponding cell-type prototypes. However, this could potentially disturb the model’s generalization capabilities because it may introduce a bias against acquiring new cell types. An alternative conceivable solution is to implement contrastive-based loss [ 37 , 38 ], represented as

The contrastive-based loss integrates current and previous samples into one batch and assesses the similarities of anchor-to-sample pairs. Here, |$J(x)$| symbolizes the index set of samples, excluding anchor |$x$| in the same batch, whereas |$ a \in A(x)$| represents the set of samples with identical labels to the anchor |$x$|⁠ . Unlike softmax-based loss, the selected pairs do not depend on the number of cell types, they are instead related to the number of samples in a training batch. Consequently, its effectiveness is restricted by the dimensions of the memory buffer and batch size. In scenarios with only a few samples for replay, its performance may not be satisfactory. To tackle these issues, we suggest employing prototype-based contrastive learning. This approach could prevent the occurrence of catastrophic forgetting by combining the benefits of both prototype- and contrastive-based learning methods. More specifically, we substitute the samples of anchor-to-sample pairs with prototypes in contrastive-based loss. In the |$s$| -time period, for instance, our coupling-inspired prototype-based contrastive loss is defined as

where |$\mathcal{C}_{B}^{s}$| represents the cell type indices in the current training batch at the |$s$| th stage, with the indices being potentially repeated. By leveraging prototypes, this loss can handle small sets of exemplars, thus eluding the limitation imposed by memory buffer size. Moreover, the substituted prototypes are derived only from the cell types present in the training batch, ensuring that propagation gradients originate solely from the learning of these cell types. In each learning iteration, only the new and old cell types in the current batch contribute to gradient propagation. The prototypes of old cell types, which are influenced by the negative gradient of new cell types, have the capacity to generate a positive gradient for conflict resolution and additionally alleviate the forgetting issue. In conclusion, the integration of contrastive learning and prototype learning can yield an optimal solution for incremental cell-type annotation.

Cell-type decorrelation learning

While we manage to mitigate catastrophic forgetting to a certain extent by calibrating gradient propagation across new and old cell categories, we contend that refining the embedding representations at each phase is paramount to enhancing the model’s efficacy. All data can be concurrently processed in joint learning, ostensibly allowing the taught representations of every cell category to be evenly distributed within the embedding space. In stark contrast, a model trained with a limited number of cell categories tends to arrange the representations in a long, linear region, generating a more evenly dissected representation with an expanding number of training cell categories [ 39 , 40 ]. Our preference is towards applying model regularization to yield scattered representations akin to those found in joint learning at every stage. The zenith of our task might be joint learning as it advocates for simultaneous data processing. We propose that if collectively learned representations of each cell type are uniformly spread within the embedding space, mimicking its representation could enhance performance at the present stage while simultaneously rendering the current stage representations more adaptable for incremental processing of new cell categories.

Stemming from our motivation, we introduce the concept of Global Partition Measurement (GPM). This concept is designed to distinctly separate samples of varying cell types following a learning model that identifies the most secure partitioning method. To further extend on this, consider an example where the stage of operation is |$s$| - in this context, the training data can be presented as |$\mathcal{X}^{sr*}=\{(x_{j}^{sr},y_{j}^{sr})_{j=1}^{n_{sr}}\}\cup \bigcup _{i=1}^{k_{1:s-1}}E_{i}^{s}$|⁠ . Accordingly, we can then determine the prediction probability for each sample |$x_{i}$| at the |$s$| th stage, represented as |$p_{s}^{i} = (p_{s1}^{i}, p_{s2}^{i},..., p_{sk_{1:s}}^{i})^{T} \in \mathcal{R}^{k_{1:s} \times 1}$|⁠ . Here, |$k_{1:s}$|⁠ , which equals |$\sum \limits _{i=1}^{s}k_{i}$|⁠ , corresponds to the total number of cell types learned after the |$s$| th phase. From this, we can derive the prediction matrix for all |$n_{sr*}$| samples, as shown in

For ease of interpretation, we reference the |$j$| th row of |$p_{s}$| in

Given that |$q_{sj}$| accumulates the probability values of all samples corresponding to the |$j$| th cell type, it can be termed the cell type assignment vector’, illustrating the broader assignment statistics of that specific cell type. For ensuring a homogenous distribution of data across each cell type, we posit that any two vectors, |$q_{sa}$| and |$q_{sb}$|⁠ , should ideally be orthogonal, implying a cosine similarity of zero, as represented in

In accordance with the preceding analysis, we describe the GPM during the |$s$| th stage as the cosine similarity set amounting to all the pairs of cell-type assignment vectors, as shown in

Here, |$G_{s}$| manifests as a |$k_{1:s} \times k_{1:s}$| matrix. Our objective revolves around reducing the GPM values, excluding diagonal elements and facilitating maximal separation amongst various cell types, thus ensuring a uniform dispersion in the embedding space. This training aim bolsters self-attention by viewing each cell type as a standalone data sample while diminishing attention between samples. Therefore, we apply the softmax operation to each cell type |$a$|⁠ , obtaining a probability measurement, as represented in

Subsequently, we administer the loss of cross-entropy, considering |$g_{s}(a, a)$| as the model prediction probability aligned with a training sample’s ground-truth label, illustrated in

By formulating |$L_{cwd}^{s}$|⁠ , we can effectively curtail non-diagonal elements of the GPM matrix, |$G_{s}(a, b)$|⁠ , thereby ensuring a uniform representation of different cell types.

In conclusion, for the |$s$| -stage, we incorporate the data denoising loss |$L_{zinb}^{s}$| to provide the comprehensive training objective, represented as

where |$\lambda $| and |$\gamma $| are loss weight hyperparameters.

Dataset composition

To encompass various potential scenarios for incremental learning of cell types, we categorize our experiments into three distinct sections: intra-data, inter-tissue and inter-data. It should be noted that batch effects are present between training and testing data in the latter two categories. Each category includes a selection of large-scale, atlas-level datasets with a substantial imbalance in cell-type composition. Our choice for the intra-data category includes the datasets of Cao, featuring 16 cell types and 32 061 cells [ 41 ], Quake 10x with 36 cell types and 54 967 cells [ 42 ], Quake Smart-seq2 comprising 45 cell types and 44 807 cells [ 42 ], and Zeisel encompassing 21 cell types and 160 796 cells [ 43 ]. Unless otherwise specified, we divide their cell types into four exclusive subcategories accounting for the cell types available at each stage. The training and testing sets are then apportioned based on a 1:9 ratio, which results in a labeled ratio of 0.1. For the inter-tissue category, the Cao_2020 atlas provides us with four tissues [ 44 ]: Eye with 11 cell types and 51 836 cells, Intestine with 12 cell types and 51 650 cells, Pancreas with 13 cell types and 45 653 cells, and Stomach consisting of 10 cell types and 12 106 cells. A degree of overlap is observed between cell types in paired tissues. These tissues are trained in alternating alphabetical orders at various stages. Lastly, for the inter-data segment, we utilize four datasets from different tissues that have been sequenced on distinct platforms. Specifically, we use He with 11 cell types and 15 680 cells [ 45 ], Madissoon with 17 cell types and 57 020 cells [ 46 ], Stewart with 18 cell types and 26 628 cells [ 47 ], and Vento containing 17 cell types and 64 734 cells [ 48 ]. Again, these are alternately trained in alphabetical order. Additionally, a split labeled ratio of 0.1 is maintained for the training and testing data in both the second and third categories. For complete details regarding these datasets, please refer to the SI.

Evaluated baselines

Our methodology targets the resolution of catastrophic forgetting problems occurring in cell-type incremental annotation learning—a niche befitting to our approach due to an absence of established annotation baselines. In a bid to demonstrate the effectiveness of scEVOLVE, we juxtapose it against four distinct methodological categories. The preliminary category utilizes an individual training strategy. This strategy is based on the training of the model exclusively on the data from the current stage. The exemplified training loss for this strategy can be numerically defined as |$L_{zinb}^{s} + \lambda L_{ind}^{s}$|⁠ , with the performance levels serving as the fundamental baseline for our task. The second often-employed strategy is the joint learning approach, using all data gathered thus far for training the model. The loss encountered in joint learning could be quantified as |$L_{zinb}^{s} + \lambda L_{joi}^{s}$|⁠ , and the corresponding performance serves as the upper-efficiency boundary for our task. Thirdly, with an aim to illustrate the influence of sample replay, the current data, alongside the exemplars, employ the cross-entropy loss, represented as |$L_{zinb}^{s} + \lambda L_{rep}^{s}$|⁠ . In light of this, we substituted |$L_{rep}^{s}$| with |$L_{pbc}^{s}$|⁠ , which is represented as |$L_{zinb}^{s} + \lambda L_{pbc}^{s}$|⁠ , to monitor the improvements resulting from replay based on prototype-based contrastive methods. Subsequently, integrating cell-type decorrelation regularization into scEVOLVE alludes to an examination of how this principle could impact performance. To ensure that the modules put forth in our proposed methodology could be seamlessly integrated into existing single-cell annotation procedures, we selected three exemplary methods: scmap hinged on traditional statistical learning [ 11 ], scANVI leveraged on deep representation learning [ 13 ] and CIForm based on transformer structure [ 49 ]. Considering that scmap and scANVI do not rely on the deep softmax classification mechanism, only the sample replay module can be integrated into their implementational design. Contrarily, all three proposed modules are compatible with CIForm.

Evaluation metrics

We evaluate the performance of all methods by reporting their classification accuracy. The term Old Accuracy pertains to the methods'classification accuracy derived from the testing data from all preceding stages. New Accuracy signifies the methods'classification accuracy as based on the testing data culled from the current stage. Moreover, Overall Accuracy is the summative classification accuracy on the collective data set, which includes testing data from both previous and current stages. The accuracy values that we report represent the average value culled from three separate runs.

Implementation details

Our algorithm implementation uses PyTorch and our experiments are conducted using two Tesla A100 GPUs. The encoder structure consists of two layers, sized 512 and 256, respectively. In contrast, the decoder possesses a structure that mirrors the encoder. The bottleneck layer carries a dimension of 128 while the training mini-batch size is set at 256. Optimization is achieved through Adam, having a learning rate set at 1e-4. Each cell type’s default replay number, denoted by |$m$|⁠ , is set to 20 and a temperature parameter |$\tau $| is held constant at 1.0. The two loss weight parameters are denoted |$\lambda $| and |$\gamma $|⁠ , with both being set at 1.0. During the initial stage, the network is warmed up with |$L_{zinb}^{s}$| over a period of 200 epochs. This is coupled with other classification loss to finetune the network for an additional 200 epochs. Subsequent stages utilize the checkpoint from the terminating stage to initiate the model, and optimization is conducted using suitable losses spanning a total of 200 epochs.

For ease of typesetting, we have incorporated the experimental results of the subsequent benchmarks into the SI.

Intra-data benchmark

Old accuracy comparison.

We initiate our analysis by benchmarking scEVOLVE against four distinct baselines using three different classification accuracy metrics across four real datasets: Cao, Quake 10x, Quake Smart-seq2 and Zeisel. The performance of each model, as depicted in Table 1 , shows a diminishing trend in accuracy as the stages increase. Notably, individual learning exhibits the most significant performance deterioration, pointing to marked catastrophic forgetting issues. However, scEVOLVE demonstrates minimal impact when incorporating incremental annotation, in comparison to other baselines—with the exception of joint learning. This underscores scEVOLVE's effective algorithmic strategies in combatting forgetting. Notably, the ordinary replay approach, exclusively utilizing cross-entropy loss with exemplars, is insufficient to prevent catastrophic forgetting, resulting in limited old accuracy. Such outcomes can be attributed to the significant discrepancy in exemplar sample size and a potential imbalance in gradient propagation based on current data, leading to catastrophic forgetting. Yet, contrastive replay performs better than ordinary replay, indicating that our proposed prototype-based contrastive learning potentially remedies the imbalance problem. Despite its advantage, contrastive replay still falls short when compared to scEVOLVE, suggesting the significance of uniformly scattered representations and the necessity for the model to generate representations akin to those of joint learning to tackle catastrophic forgetting for optimal performance. As the model learns more cell types, it becomes increasingly challenging to annotate each cell-type sample, resulting in a slight but inevitable decrease in joint learning's old accuracy. Based on our analysis, scEVOLVE ranks second only to joint learning in terms of old accuracy, showcasing its ability to retain previously learned cell types effectively.

Comparative analysis of performance among diverse baselines in intra-data incremental annotation benchmarking. The term ‘OR’ denotes Ordinary Replay, while ‘ALL’ is indicative of Prototypical Contrastive Replay and Cell-Type Decorrelation

scmap, scANVI and CIForm are not originally designed for incremental annotation and only function optimally within fixed cell-type knowledge parameters. Hence, applying these approaches to our benchmark could induce a deterioration in the performance of old accuracy, akin to individual training results. However, a significant improvement in their performance can be observed, particularly for scmap and CIForm, when the simple replay module is integrated, underscoring the efficacy of the sample replay strategy. Although enabling scANVI to handle the incremental annotation task is achieved by this addition, the performance of scANVI bolstered by the ordinary replay strategy remains underwhelming, highlighting the need for the development of more customized algorithms. A comparative assessment of CIForm with ALL and CIForm with OR testifies to the capacity of our comprehensive strategies to more effectively aid CIForm in overcoming the catastrophic forgetting challenge, thus demonstrating its superior performance. Consequently, our proposed modules can be readily incorporated into single-cell annotation methods predicated on deep softmax classification.

New accuracy comparison

The analytical efficacy of scEVOLVE demonstrates superior functionality in terms of annotating current data, surpassing joint learning, but not as competitive when juxtaposed with the other three methodologies. This outcome is anticipatable owing to each method's distinct balance between retaining knowledge of old cell types and assimilating new cell types. As part of its intentional design strategy, scEVOLVE tolerates marginal new accuracy loss to mitigate the issue of catastrophic forgetting. Notably, the anti-forgetting strategy of scEVOLVE can be applied to most existing softmax-based annotation methods through the integration of relevant loss functions. In contrast, joint learning concedes greater new accuracy to retain the memory of old cell types by repetitively learning previously studied samples. Overloading its learning cohort leads to an ultimate loss of focus on acquiring new cell types, thus significantly impeding its new accuracy performance. With the addition of multiple training stages, joint learning also incurs substantial costs due to numerous from-scratch’ training sessions involving complete data sets. With new data constantly streaming in, this method of retraining the network each time is unsustainable. On the other hand, individual learning, while offering relatively satisfactory results in annotating new cell types, fails to address the catastrophic forgetting problem, rendering it impractical for the incremental annotation scenario we propose. Likewise, both ordinary replay and contrastive replay offer competitive performance concerning new accuracy. However, their less-than-optimal old accuracy results make them less desirable choices.

Additionally, incorporating traditional replay tactics or our suggested comprehensive strategies into other existing single-cell annotation methods could potentially diminish their new accuracy on the existing dataset. This is reasonably anticipated, given that the storage and replay of old samples can exacerbate the challenge of learning from new samples. However, the application of incremental annotation mechanisms can still enable these methods to yield reasonably competitive results in terms of new accuracy. This suggests that the adverse effect introduced by old samples remains insignificant.

Overall accuracy comparison

The analysis conducted illustrates the necessity of striking a balance between old and new accuracy assessments while employing overall accuracy as the primary evaluation criterion. Notable findings demonstrate that scEVOLVE consistently yields satisfactory outcomes, particularly in the late stages. Despite a potential minor loss of precision in identifying new cell types, scEVOLVE maintains impressive performance levels across all cell types. This signifies an effective compromise between preserving knowledge about old cell types and acquiring proficiency in discerning new ones. On the one hand, joint learning naturally achieves greater overall accuracy as an upper limit compared to other methodologies. However, the marginal difference between the overall accuracy of scEVOLVE and joint learning exceptionally validates scEVOLVE's efficacy. On the other hand, individual learning's limited capacity in dealing with old cell types makes its overall accuracy less competitive, setting the lower limit. Regular and contrastive replay methods fall short in terms of overall accuracy due to their mediocre performance on old cell type's accuracy. This implies an unfavorable trade-off in these approaches. A critical insight gained from comparing scEVOLVE with contrastive replay was the paramount importance of the cell-type decorrelation process. This strategy enhances the segregation of samples from diverse cell types by ensuring homogeneous scattering of each cell-type representation, thereby contributing to mitigating the issue of memory erosion. Comparisons between scEVOLVE and the overall accuracies of other annotation methods utilizing OR and ALL further indicate subpar outcomes. This suggests inherent differences in the ability of varied baseline models to capture the data structure. Contrastive replay methods and cell-type decorrelation strategies, as employed in CIForm with ALL, manifestly yield more beneficial results than ordinary replay like CIForm with OR. In summary, in terms of intra-data experiments, scEVOLVE outperforms other baselines by considerable margins and effectively circumvents the severe issue of catastrophic forgetting during the incremental annotation of cell types.

Visualization comparison

To unequivocally illustrate the differentiation prediction of various cell types, we have extensively utilized Sankey plots. These plots provide a visual representation of annotation results at the fourth stage of scEVOLVE, as well as other baselines on the Zeisel dataset, as depicted in Figure 2 . Our proprietary scEVOLVE system accurately identifies correspondences for both previously known and newly discovered cell types. This substantiates our assertion that our devised tactics effectively impede catastrophic forgetting, while concurrently maintaining superior performance capacity in learning novel cell types. We can further infer that scEVOLVE's overall annotation performance is on par with joint learning, a benchmark for memory retention problems. This is yet another testament to scEVOLVE's significant advancement in addressing incremental annotation challenges. Despite this, the performances of individual and ordinary replay learning methods remain suboptimal. For instance, individual learning tends to forget numerous established cell types, consequently misassigning them to newly identified cell types. Conversely, ordinary replay inaccurately identifies a certain percentage of actrocyte cells as radial glial cells. This discrepancy likely stems from significant differences in the quantity of exemplars and current samples, ultimately leading to a learning imbalance between old and new cell types. This predisposes the system to categorize cells as new types. The observations further attest to the supremacy of scEVOLVE, underlining its efficacy and reliability in predicting diverse cell types.

(A–D) The Sankey mapping plots of test data in the fourth stage for different baselines on the Zeisel dataset. In each plot, the left column represents the true cell type, and the right column represents the predicted cell type. Specific cell-type information can also be seen in the SI.

( A – D ) The Sankey mapping plots of test data in the fourth stage for different baselines on the Zeisel dataset. In each plot, the left column represents the true cell type, and the right column represents the predicted cell type. Specific cell-type information can also be seen in the SI.

To enhance the grasp of scEVOLVE's annotation results pertaining to old and new cell types, we have visualized the low-dimensional portrayals of scEVOLVE utilizing t-distributed stochastic neighbor embedding (t-SNE) [ 50 ]. This is demonstrated in Figure 3 which presents the t-SNE plots of the testing data across four distinct stages for scEVOLVE on the Zeisel dataset. Each graph set's left subgraph illustrates the classification results based on the understood truth label, while the right subgraph exhibits the classification outcomes stemming from the predicted label. A noteworthy observation is the striking similarity between the classification diagram derived from the predictive label and those from the true label at every stage. This reveals scEVOLVE's capacity to predict the samples with impressive accuracy. Notably, as the training stage evolves, scEVOLVE's discriminatory ability between diverse cell types, both old and new, becomes increasingly evident. This supports scEVOLVE's capability to achieve a balance between enhancing accuracy for previously established types and preserving accuracy for newer types. Furthermore, defined dividing lines are observable between different cell types in the embedding space, a phenomenon that could be attributed to the proposed cell-type decorrelation, aimed at improving the embedding representations, hence aiding cell-type annotation. Overall, it can be asserted that scEVOLVE exhibits outstanding performance with respect to both old and new accuracies within the intra-data experiments.

(A–D) The t-SNE visualization plots of test data in four stages for scEVOLVE on the Zeisel dataset. In each plot, the legend represents a set of cell types, and different colors represent different cell types. Specific cell-type information can also be seen in the SI.

( A – D ) The t-SNE visualization plots of test data in four stages for scEVOLVE on the Zeisel dataset. In each plot, the legend represents a set of cell types, and different colors represent different cell types. Specific cell-type information can also be seen in the SI.

Inter-tissue benchmark

Three kinds of accuracy comparison.

In the subsequent steps, we conducted a series of tests utilizing actual data sets extracted from a variety of tissue types, namely, the Eye, Intestine, Pancreas, and Stomach. These datasets were chosen from a collection of atlas data. The task presented a formidable challenge due to the necessity of grappling with extensive cross-tissue data that possessed a slight batch effect. This required high performance from the algorithm to efficiently process data including striking variances in sample sizes between distinct cell types and the elimination of the batch effect.

Insights from Table 2 demonstrate that scEVOLVE possessed commendable performance in terms of annotation accuracy concerning old cell types; its results were only marginally surpassed by those that utilized the joint learning approach. Interestingly, scEVOLVE emerged as the best approach in achieving the utmost precision concerning new cell types, outperforming all other baseline methods. One of the reasons for this superior performance was that scEVOLVE was adept at shrinking the requirement of replaying old sample sizes. This grants it a higher degree of flexibility to concentrate on new cell types. Notably, scEVOLVE proved to be more efficient in handling large-scale data as compared to smaller-scale data. The accuracy of other methodologies, in comparison, dropped significantly. This can be attributed to its prototype-based contrastive learning, which adeptly balances the gradient propagation of both new and old cell types. This helps to address the problem of cell-type imbalance effectively. It further regularizes embedding representations to scatter uniformly which can separate different cell types completely. On the other hand, joint learning is impeded by the sheer volume of data it has to deal with. This directly hinders its learning capacity concerning new cell types, hence leading to an inadequate level of new accuracy. Furthermore, joint learning is also taxing in terms of memory requirements for handling large-scale data. This places considerable pressure on hardware configurations, hence reducing its competitiveness in practical applications. While comparing these performances with scEVOLVE, we noted that the removal of prototypical contrastive learning or cell-type decorrelation would negatively impact overall performance. In essence, this emphasized the importance of these two proposed strategies. In conclusion, given considerations such as remembering and learning new or old cell types, managing large-scale, tissue-based sequential data, and varying learning orders, scEVOLVE showcased robust and superior performance. This underlines its potency in effectively handling inter-tissue incremental annotation benchmark tests.

Comparative analysis of performance among diverse baselines in inter-tissue incremental annotation benchmarking. The term ‘OR’ denotes Ordinary Replay, while ‘ALL’ is indicative of Prototypical Contrastive Replay and Cell-Type Decorrelation

Inter-data benchmark

In this section, we delve deeper into the performance of scEVOLVE and four other benchmark algorithms across four major real datasets in the inter-data investigations. These real datasets are derived from various tissue types, denoted as He, Madisoon, Stewart and Vento. They emanate from diverse atlas data, utilizing different sequencing technologies, and consequently, engender a somewhat augmented batch effect. Analyzing these datasets poses a considerable challenge due to the intricate batch effect that heightens the performance demands of the algorithms. Table 3 presents a level of complexity reflected in the performance of all methods as a result of the batch effect. Notwithstanding, scEVOLVE upholds its high-quality performance, signifying its potential to coherently map cells from contrasting datasets within a single latent space. Furthermore, judging by the old accuracy, its faculty to prevent learning loss remains unaffected by the batch effect, illustrating its robustness. It's plausible that prototype-based contrastive learning could assist in attenuating distribution bias between datasets simplicity by aligning each sample to its own prototypes. On the downside, joint learning has evidently lost some of its competitive edge, particularly in new accuracy. This is due, in part, to its inability to mitigate the batch effect, in addition to having to be retrained on increasingly sizeable datasets, which drastically compromises its adaptability in learning new cell types. Individual learning, because of its unique learning characteristics, is minimally impacted by the batch effect, but its performance leaves much to be desired, especially regarding old accuracy, given its lack of capacity to avoid learning loss. In comparison to scEVOLVE, the old and new accuracy levels of ordinary replay and contrastive replay are inferior, due to the integration of prototypical contrastive learning and cell-type decorrelation. It's also worth noting that scmap's performance in combination with OR produces the least favorable results, implying that the ability to handle batch effects greatly influences the performance in the inter-data benchmark. Despite CIForm with ALL exhibiting superior performance in relation to other methods combined with OR, it still falls short of scEVOLVE's level, suggesting that merely integrating our framework with other annotation methods may not suffice. Fine-tuning specific parameters might be necessary for various basic models. Therefore, scEVOLVE distinguishes itself from other benchmark algorithms through its capacity to effectively counteract the batch effect, making it an ideal and highly competitive solution in real-world scenarios.

Comparative analysis of performance among diverse baselines in inter-data incremental annotation benchmarking. The term ‘OR’ denotes Ordinary Replay, while ‘ALL’ is indicative of Prototypical Contrastive Replay and Cell-Type Decorrelation

To more clearly demonstrate scEVOLVE’s annotation results and feature representations, a t-SNE visualization plot is presented in Figure 4 following the fourth stage. The initial and subsequent rows in the same figure depict the annotative outcomes from individual learning and scEVOLVE respectively. Significantly, the leftmost image, demonstrating the division of samples across various domains, presumably indicates the varying capabilities of different methods to eliminate batch effects. A comparison reveals scEVOLVE's superior ability in distinguishing data from different domains in contrast to individual learning, which tends to amalgamate them, particularly evident in the case of Madissoon and Vento. This attribute of scEVOLVE may stem from its capacity to harmonize samples with their prototypes and to enhance the embedding of representations. These attributes further assist the algorithm in learning a more effective latent space and effectively eliminating the batch effect. The central and right-sided images represent the two-dimensional visualization plots drawn from the ground-truth labels and prediction labels correspondingly. scEVOLVE demonstrates a clear capability to differentiate between varying cell types. This primarily owes to its regularization of embedding representations to scatter uniformly at every stage and its ability to maintain a balance in the gradient propagation. These features help the model in recalling and annotating samples of previous cell types. Conversely, the latent space evolved from individual learning fails to cultivate a proper and unique cluster structure, thereby vividly demonstrating catastrophic forgetting. Thus, it can be concluded that scEVOLVE effectively addresses this problem of forgetting and provides superior performance overall.

(A, B) The t-SNE visualization plots of test data in four stages for individual training and scEVOLVE on sequential He, Madissoon, Stewart, and Vento datasets. In each plot, the legend of the left subgraph represents the dataset set, and the middle subgraph and the right subgraph share the same legend, representing the cell-type set. Specific cell-type information can also be seen in the SI.

( A , B ) The t-SNE visualization plots of test data in four stages for individual training and scEVOLVE on sequential He, Madissoon, Stewart, and Vento datasets. In each plot, the legend of the left subgraph represents the dataset set, and the middle subgraph and the right subgraph share the same legend, representing the cell-type set. Specific cell-type information can also be seen in the SI.

Robustness analysis

Accuracy variation in inter-tissue benchmark.

In order to evaluate the resilience of various procedures when confronting the variability of data learned at different stages, we reconfigured the sequence of large-scale data learned in the four stages and assessed the performance of scEVOLVE and additional benchmarks. The corresponding experimental results from datasets within the inter-tissue benchmark are presented in Figures 5 and 6 . Notably, individual learning displays significantly lower performance on old accuracy compared to other methods, thus, to make the comparative performance evaluation of other methods more distinct, it was excluded from Figure 5 . For clarity in presentation, the following terms are used in the legends of the figures: individual, ‘ordinary’, ‘contrastive’ and ‘joint’ to represent individual learning, ordinary replay, contrastive replay and joint learning, respectively. It is observed that the overall performance of all methods tends to be less stable regarding new accuracy, in comparison to old accuracy. This can be primarily attributed to the fact that old cell types were learned before the finalization of the learned embedding space and the corresponding parameters that might directly influence the learning process of the new cell type. As depicted in Figures 5 and 6 , the performance of scEVOLVE remains relatively stable for both old and new accuracy, irrespective of the learning sequence variations of datasets, demonstrating its robustness. scEVOLVE's embedding representation mimics that of joint learning, hence, it remains rather insulated from the learning sequence of the data. Additionally, its strategy to prevent memory fading by managing gradient propagation aims to balance the learning of both old and new cell types, thereby promising equal treatment by the model for both. In contrast, violent fluctuations are observed in the old accuracy of individual learning and the new accuracy of joint learning, underscoring their instability. The performance of ordinary replay also has shortcomings, indicating that merely reducing memory fading by storing exemplars is a weak strategy that leaves its accuracy easily susceptible to changes in the learning order of the dataset. Similarly, contrastive replay often provides sub-optimal results, being too dependent on the learning order of datasets due to a lack of stability control in representation learning; the embedding features of the current stage are altered along with the variation of features from the previous stage. In summary, scEVOLVE consistently showcases robust performance notwithstanding different sequencing of datasets in the inter-tissue benchmarks.

Line graph of the variation in old accuracy of the four methods when the four datasets from different tissues are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’.

Line graph of the variation in old accuracy of the four methods when the four datasets from different tissues are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’.

Line graph of the variation in new accuracy of the five methods when the four datasets from different tissues are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, ‘individual’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’.

Line graph of the variation in new accuracy of the five methods when the four datasets from different tissues are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, ‘individual’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’.

Accuracy variation in inter-data benchmark

In an effort to examine the model's resilience to alterations in the learning sequence of datasets under the batch effect, we evaluated the annotation accuracy fluctuations of scEVOLVE in comparison to other methods using the inter-data benchmark. Figures 7 and 8 demonstrate the trajectories of both old and new accuracy across all analyzed methods. We chose not to graph the old accuracy line for individual learning due to its inferior performance. Primarily, we observed that the old and new accuracy of all methods exhibited greater instability compared to those displayed in the inter-tissue benchmark. This is to be expected, given that the pronounced batch effect presents a more significant obstacle to the incremental learning of models. In parallel to occurrences witnessed with minor batch effect incidents, scEVOLVE consistently yielded stable findings, as displayed in Figures 7 and 8 . Subsequently, despite the presence of notable batch effects, these findings reveal that scEVOLVE preserves satisfactory and robust old and new accuracy across the compared methods, underscoring its effectiveness and resilience in counteracting batch effects. In contrast, joint learning's new accuracy performance was the weakest amongst all the methods, showcasing violent fluctuations. This could be ascribed to the necessity of retraining all samples to acquire new cell types at every stage, leading to heightened sensitivity to the learning order of the dataset. Nonetheless, while ordinary replay and contrastive replay demonstrated stability akin to scEVOLVE, their performance was inferior when matched against scEVOLVE's aptitude for controlling gradient propagation and standardizing embedding representation, both of which are paramount for reminiscing old cell types and assimilating new ones. In summary, when juxtaposed with other baselines, scEVOLVE can effectively cushion the impact of batch effects. Furthermore, varying the learning sequence of the dataset has minimal influence on its performance regarding both old and new accuracy.

Line graph of the variation in old accuracy of the four methods when the four datasets with batch effects are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ordinary’ and ‘scEVOLVE (ours)’.

Line graph of the variation in old accuracy of the four methods when the four datasets with batch effects are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ordinary’ and ‘scEVOLVE (ours)’.

Line graph of the variation in new accuracy of the five methods when the four datasets with batch effects are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, individual’, ‘joint’, ordinary’ and ‘scEVOLVE (ours)’.

Line graph of the variation in new accuracy of the five methods when the four datasets with batch effects are learned at different stages. In each graph, the legend represents the set of comparison methods, namely ‘contrastive’, individual’, ‘joint’, ordinary’ and ‘scEVOLVE (ours)’.

Discussion of hyperparameters

Sensitivity analysis for exemplar size.

In this section, we explore the impact of individual hyperparameter values on the performance of the model. We initially turn our attention to |$m$|⁠ , a hyperparameter that dictates the quantity of exemplars retained for each cell type. This hyperparameter plays a crucial role in the performance enhancement of scEVOLVE. Increasing |$m$| has a dual effect. On the positive side, a higher value of |$m$| lets the exemplars encapsulate a richer array of data pertaining to established cell types, thereby elevating the model's accuracy in relation to these old cell types. Conversely, a higher |$m$| value also imposes a greater computational strain on the model, increasing both the required learning capacity and computational resources. This necessitates the model to allocate additional time and memory for mastering larger exemplar sets, conversely depriving it of the time to comprehend new cell types. This inevitably leads to a drop in accuracy when encountering new cell types. Consequently, choosing the optimal |$m$| value requires a careful balance between retaining knowledge about established cell types and accommodating the learning of new cell types. This underscores the necessity of analyzing the impact of |$m$| on the model's precision.

In our study, we directed investigations on two distinct datasets: Quake 10x and Quake Smart-seq2. The changing patterns of the overall precision of scEVOLVE at the fourth stage are visually represented in both Figures 9(A) and 9(B) . To focus on instances where the increase in |$m$| oscillates between the spectrum of |$[10,15,20,25,30]$|⁠ , we observed a consistent rise in the total precision of each method as the value of |$m$| escalated. This infers that assessing and calibrating the model's accuracy against its computational load must be carried out with practical applications in mind. Particularly when |$m$| manifests a smaller value, for example, 10, scEVOLVE is competent in delivering superior performance. This reinforces its dominance in maintaining equilibrium between the education of established and emerging cell types. While ordinary replay and contrastive replay's executions uplift with the growth of |$m$|⁠ , the outcome is not as optimal as scEVOLVE. As a general rule, we regularly established |$m$| at 20. This approach has proven effective in ensuring a balance between precision and computational load, and it has reinforced the superior performance of the scEVOLVE method in our experiments.

Overall accuracy on Quake 10x and Quake Smart-seq2 datasets. (A, B) Changing the number of exemplars for each cell type, and the legend represents the set of comparison methods, namely ‘contrastive’, ‘ordinary’ and ‘scEVOLVE (ours)’; (C, D) changing the labeled ratio, and the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’.

Overall accuracy on Quake 10x and Quake Smart-seq2 datasets. ( A , B ) Changing the number of exemplars for each cell type, and the legend represents the set of comparison methods, namely ‘contrastive’, ‘ordinary’ and ‘scEVOLVE (ours)’; ( C , D ) changing the labeled ratio, and the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’.

Sensitivity analysis for labeled ratio

The hyperparameter known as the labeled ratio in the model governs the proportion of data designated for training and testing. Essentially, a greater labeled ratio equates to a larger quantity of data employed as training data within the dataset. From this, we extrapolate that an elevated labeled ratio could potentially enhance new accuracy as it facilitates deeper learning of the existing training data relevant to the current cell type. However, this scenario might inversely affect the old accuracy. An increase in training data may intensify the inequity between the model's comprehension of new and old cell types, thus impeding its capability to recall old cell types. Consequently, it becomes imperative to investigate the effects of varying labeled ratios on the model's precision to optimize its accuracy.

Figure 9(C) and (D) distinctly illustrates the shifting trends of overall precision during the fourth phase of scEVOLVE and other baseline parameters when the labeled ratio alters on Quake 10x and Quake Smart-seq2 datasets, respectively. Our analysis synthesizes instances where the incremental change of the labeled ratio spans within the scope of |$[0.05,0.075,0.1,0.125,0.2]$|⁠ . Observations suggest that the integral accuracy of scEVOLVE remains relatively unwavering in relation to the labeled ratio, underscoring that our model is only minimally influenced by the magnitude of the labeled ratio. Importantly, scEVOLVE consistently exhibits commendable performance compared to other studied methods, thereby validating its preeminence in curtailing the oblivion of antique cell types while facilely learning novel cell types. A slight elevation in the comprehensive accuracy is noticeable in the context of joint learning as the labeled ratio surges, signifying its growing need for an expanded body of training data. Contrastingly, the performance trajectories of both ordinary and contrastive replay display a clear decrement as the labeled ratio heightens. This suggests that an enlarged set of training data may augment the difficulty of equilibrating the learning process between conventional and contemporary cell types. For the purposes of this paper, we have defaulted the labeled ratio to 0.1.

Sensitivity analysis for stage number

It has been established that multiple learning stages in the model could potentially increase the propensity to forget old cell types. Similarly, integrating excessive cell types for learning purposes might compromise the model's capacity to acquire new cell types. Consequently, it is imperative to examine the performance of scEVOLVE and other fundamental benchmarks in relation to an expanded number of stages. Empirical experiments have been performed on two large-scale, real data sets, Quake 10x and Quake Smart-seq2, featuring nine learning stages. Figure 10(A) and (B) , respectively, illustrates the alterations in total accuracy across five methodologies related to these data sets. However, as individual learning significantly lags due to its inherent inability to prevent cell-type forgetting, these trends have not been graphically represented for streamlined observation. Specifically, under these stringent conditions, a noticeable downward shift in overall accuracy across all methodologies was observed, aligning with our assumption of increased difficulty in preventing forgetting old cell types while assimilating new ones as stage count increases. Amongst various models, joint learning was the most resilient, demonstrating its dominance over incremental learning associated with a higher number of stages, despite its declining competitiveness due to memory demand saturation. The moderation in variation trend displayed by scEVOLVE, although secondary to joint learning, further corroborates its efficacy in averting catastrophic forgetting and assimilating new cell types, even amidst escalating stages of incremental learning. Comparatively, the decrement trend of both ordinary replay and contrastive replay is quite abrupt, rendering them less efficient than scEVOLVE considering its superior capacity to regulate gradient propagation and standardize embedding representation. To summarize, scEVOLVE can efficiently withstand the load of extra stages, thus demonstrating its eligibility and promising performance, making this model a pragmatic choice for real-world situations.

(A, B) The variations of overall accuracy in nine stages on Quake 10x and Quake Smart-seq2 datasets, and the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’; (C, D) the overall accuracy variations of scEVOLVE with the changes of temperature $\tau $, weight $\lambda $ and $\gamma $ on the Cao, Quake 10x and Quake Smart-seq2 datasets in the last (fourth) stage. The legend in (c) represents the set of tested datasets.

( A , B ) The variations of overall accuracy in nine stages on Quake 10x and Quake Smart-seq2 datasets, and the legend represents the set of comparison methods, namely ‘contrastive’, ‘joint’, ‘ordinary’ and ‘scEVOLVE (ours)’; ( C , D ) the overall accuracy variations of scEVOLVE with the changes of temperature |$\tau $|⁠ , weight |$\lambda $| and |$\gamma $| on the Cao, Quake 10x and Quake Smart-seq2 datasets in the last (fourth) stage. The legend in (c) represents the set of tested datasets.

Sensitivity analysis for temperature and loss weight

To demonstrate the extent of sensitivity that scEVOLVE possesses towards temperature variations denoted as |$\tau $|⁠ , we executed a series of control experiments using Cao, Quake 10x and Quake Smart-seq2 datasets. The outcomes of these experiments have been visually represented in Figure 10(C) . Throughout a substantial range of |$\tau $| values, specifically |$\tau \in [0.1, 0.2, 1.0, 5.0, 10.0] $|⁠ , there was no significant alteration in performance exceeding 1%. This evidence suggests the robustness of scEVOLVE in relation to temperature variations denoted by |$\tau $|⁠ . Moreover, an analysis of scEVOLVE’s performance response to modifications in the loss weight parameters |$ \lambda $| and |$ \gamma $| was carried out on the Cao dataset. The results depicted in Figure Figure 10(D) reveal that the overall accuracy remains relatively consistent amidst changes in |$ \lambda \in [0.1, 0.2, 1.0, 5.0, 10.0] $| and |$ \gamma \in [0.1, 0.2, 1.0, 5.0, 10.0] $|⁠ . This affirms the stability of scEVOLVE with respect to adjustments made for the parameters |$ \lambda $| and |$ \gamma $|⁠ .

Finally, in order to facilitate other researchers to use our method, we recommend some appropriate hyperparameter choice ranges in Table 4 . In addition, we still suggest that the selection of hyperparameters should depend on specific datasets and experimental resources.

The recommended usage range of all hyperparameters in our model. ‘ratio’ and ‘stage’ refer to labeled ratio and stage number, respectively

This article introduces a novel, pragmatic and demanding task of cell-type incremental annotation in the domain of single-cell research, a task that takes into account real-world cell annotation needs. This task presents a unique challenge of only having access to data from the current stage, thereby rendering the data from prior stages unavailable. This unavailability might potentially lead to severe forgetting issues in the model. To remediate this pressing concern, the scEVOLVE, an innovative continual learning framework that excludes forgetting, is tailored. This framework is a trifold design. Firstly, a memory buffer is structured to conserve select old samples employing the nearest prototype exemplar selection mechanism, thus curtailing catastrophic forgetting. Secondly, considering the prevalent cell-type imbalance issue, the framework proposes a novel prototype-based contrastive replay loss as an alternative to the traditional cross-entropy to achieve balanced gradient propagation. Lastly, to elevate model performance from a representational perspective, the framework introduces cell-type decorrelation regularization. The motive behind this inclusion is to stimulate representations to be distributed more evenly in the associative embedding space, mirroring the principles of joint learning. It is paramount to note that this incremental learning framework is entirely independent yet complementary to the foundational concepts of most deep softmax-based cell annotation algorithms. By integrating this framework with existing algorithms, one can attain superior outcomes.

To thoroughly assess the performance of the algorithm, meticulous construction of comprehensive benchmarks and baselines was done. Explicitly, scEVOLVE was put up against four competitive methods—individual learning, ordinary replay, contrastive replay and joint learning, within three rigorous scenarios namely intra-data, inter-tissue, and inter-data. Firstly, the intra-data trials revealed the proficiency of scEVOLVE, ranking second to joint learning in terms of old accuracy, but outperforming all others in new accuracy. This underscored the effectiveness of its strategies against forgetting. Furthermore, scEVOLVE epitomized an optimal equilibrium between preventing knowledge forgetting and learning new cell types. Meanwhile, drawing from the observation that joint learning tends to yield sub-optimal results when it comes to new accuracy, it can be inferred that joint learning is not always the optimal choice, even when memory capacity is ample. Rather, incremental learning like our strategies may produce superior outcomes. This assertion is validated by instances where scEVOLVE has achieved parity in accuracy with joint learning. Furthermore, the notable enhancement in accuracy when our framework is integrated with existing algorithms, such as scmap, scANVI and CIForm, amplifies the value of our framework. This notable boost in performance encourages the expansion of our framework's application to a broader range of existing models, thereby enhancing their applicability in practical scenarios. Secondly, in the inter-tissue category, scEVOLVE exhibited increased competitiveness with mass-scale data featuring subtle batch effects, surpassing its previous small-scale data performance. Simultaneously, the efficiency of all other methods witnessed a steep and inevitable decline, consolidating scEVOLVE's potential to maneuver mass-scale data in real-world situations. It is worth highlighting that the overall accuracy of scEVOLVE surpasses that of joint learning in certain datasets. This underscores the efficacy of our approach, namely prototypical contrastive replay learning, in mitigating the batch effect. Thirdly, our inter-data trials were designed to evaluate the performance of scEVOLVE, alongside other baselines, on mass-scale data enveloped with severe batch effects. The results further distinguished scEVOLVE from other methods by a noticeable margin, thereby underlining its supremacy. Besides, the orderly arrangement of a substantial variety of cell types on a two-dimensional plane further demonstrates that our cell-type decorrelation learning strategy can enhance a more uniform representation of these cell types. Fourthly, scEVOLVE's robustness and choice of hyperparameter values were examined in added experiments. The results indicate that scEVOLVE exhibits superior stability due to its ability to effectively manage data diversity at various learning stages and hyperparameter selections. This characteristic enhances its reproducibility and promotes its broader application. We have empirically validated that scEVOLVE maintains competitive performance when the training stage duration is extended from 4 to 9, further substantiating its capacity for incremental learning across a multitude of cell types over an extended timeframe. Additionally, we successfully expanded the scope of tissues studied from 4 to 8 in the SI, revealing that scEVOLVE continues to maintain high accuracy with minimal forgetting, thus preserving its efficacy. To culminate, scEVOLVE showcased an adept capability in solving the cell-type incremental annotation challenge. The implications of our findings could prove beneficial for tangible applications such as updates and upgrades of cell annotation systems.

In this paper, we pioneer the conceptualization and development of single-cell incremental annotation as a solution to the issue of catastrophic forgetting. We examine the problem within the context of an optimal close-set setting, denoted as |$\mathcal{C}^{it} \subseteq \mathcal{C}^{ir}$|⁠ . Nonetheless, the real world presents more complex scenarios such as the open-set and open-partial settings. As a result, our future work will involve enhancing our model to be adaptable to these more demanding situations. Furthermore, our approach efficiently mitigates the problem of forgetting through the storage of exemplars. However, our key aim is to create replay-free algorithms that eliminate the necessity for exemplars entirely.

While the scEVOLVE model is capable of gradually acquiring knowledge about new cell types without severely compromising knowledge of the existing types, it is necessary to address its apparent limitations. Relying solely on scRNA-seq data for cell annotation may prove deficient due to potential gaps in information or noise. As a solution, we suggest broadening the incremental annotation challenge to encompass other omics or cell morphology. This would provide a more nuanced representation of cellular heterogeneity from various viewpoints. In addition, it's worth noting that scRNA-seq data only capture a static snapshot at a specific moment in time, hence it lacks temporal information. To rectify this, we propose tailoring our task and suggested framework for time series data, which would enable us to investigate the dynamic trajectory of cell development in subsequent studies.

We introduce an innovative, feasible and demanding endeavor named cell-type incremental annotation. Additionally, we have developed an avant-garde technique, called scEVOLVE, which is skillfully tailored to execute this task with utmost efficacy.

Our scEVOLVE algorithm is engineered to preserve specific historical data within a memory buffer, an action designed for subsequent replay throughout future training phases. In an effort to effectively address the issue of cell-type imbalance, scEVOLVE innovatively establishes a prototypical contrastive learning objective.

To replicate the training of the oracle model alongside comprehensive data, scEVOLVE implements the principle of cell-type decorrelation. This principle notably enhances the uniform distribution of feature representations for each cell type.

Comprehensive studies conducted on meticulously crafted benchmarks reveal that scEVOLVE efficiently mitigates the issue of catastrophic forgetting. Furthermore, it has demonstrated the capacity to progressively learn numerous cell types over an extended duration.

This work was supported by the National Key Research and Development Program of China (2021YFF1200902) and the National Natural Science Foundation of China (32270689, 12126305).

Yuyao Zhai is a doctoral candidate at the School of Mathematical Sciences, Peking University.

Liang Chen is a senior researcher at Huawei.

Minghua Deng is a professor at the School of Mathematical Sciences, Peking University.

Kolodziejczyk AA , Kim JK , Svensson V , et al.    The technology and biology of single-cell RNA sequencing . Mol Cell   2015 ; 58 ( 4 ): 610 – 20 .

Google Scholar

Ding J , Adiconis X , Simmons SK , et al.    Systematic comparison of single-cell and single-nucleus RNA-sequencing methods . Nat Biotechnol   2020 ; 38 ( 6 ): 737 – 46 .

Lähnemann D , Köster J , Szczurek E , et al.    Eleven grand challenges in single-cell data science . Genome Biol   2020 ; 21 ( 1 ): 1 – 35 .

Yan W , Zhang K . Tools for the analysis of high-dimensional single-cell RNA sequencing data . Nat Rev Nephrol   2020 ; 16 ( 7 ): 408 – 21 .

Shao X , Liao J , Xiaoyan L , et al.    scCATCH: automatic annotation on cell types of clusters from single-cell RNA sequencing data . Iscience   2020 ; 23 ( 3 ): 100882 .

Bao W , Lin X , Yang B , Chen B . Gene regulatory identification based on the novel hybrid time-delayed method . Front Genet   2022 ; 13 : 888786 .

Bao W , Yujian G , Chen B , Huiping Y . Golgi_df: Golgi proteins classification with deep forest . Front Neurosci   2023 ; 17 : 1197824 .

Abdelaal T , Michielsen L , Cats D , et al.    A comparison of automatic cell identification methods for single-cell RNA sequencing data . Genome Biol   2019 ; 20 ( 1 ): 1 – 19 .

Wolf FA , Angerer P , Theis FJ . SCANPY: large-scale single-cell gene expression data analysis . Genome Biol   2018 ; 19 : 1 – 5 .

Chen L , Zhai Y , He Q , et al.    Integrating deep supervised, self-supervised and unsupervised learning for single-cell RNA-seq clustering and annotation . Genes   2020 ; 11 ( 7 ): 792 .

Kiselev VY , Yiu A , Hemberg M . scmap: projection of single-cell RNA-seq data across data sets . Nat Methods   2018 ; 15 ( 5 ): 359 – 62 .

Satija R , Farrell JA , Gennert D , et al.    Spatial reconstruction of single-cell gene expression data . Nat Biotechnol   2015 ; 33 ( 5 ): 495 – 502 .

Chenling X , Lopez R , Mehlman E , et al.    Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models . Mol Syst Biol   2021 ; 17 ( 1 ): e9620 .

Brbić M , Zitnik M , Wang S , et al.    Mars: discovering novel cell types across heterogeneous single-cell experiments . Nat Methods   2020 ; 17 ( 12 ): 1200 – 6 .

Lotfollahi M , Naghipourfar M , Luecken MD , et al.    Mapping single-cell data to reference atlases by transfer learning . Nat Biotechnol   2022 ; 40 ( 1 ): 121 – 30 .

Rebuffi SA , Kolesnikov A , Sperl G , Lampert CH . iCaRL: incremental classifier and representation learning . Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , p. 2001 – 10 , 2017 .

Li Z , Hoiem D . Learning without forgetting . IEEE Trans Pattern Anal Mach Intell   2017 ; 40 ( 12 ): 2935 – 47 .

Goodfellow IJ , Mirza M , Da Xiao AC , Bengio Y . An empirical investigation of catastrophic forgetting in gradient-based neural networks . arXiv preprint arXiv:1312.6211, 2013 .

Kirkpatrick J , Pascanu R , Rabinowitz N , et al.  . Overcoming catastrophic forgetting in neural networks . Proc Natl Acad Sci   2017 ; 114 ( 13 ): 3521 – 26 .

Rolnick D , Ahuja A , Schwarz J , et al.    Experience replay for continual learning . Adv Neural Inf Process Syst   2019 ; 32 :348–358.

Lopez-Paz D , Ranzato MA . Gradient episodic memory for continual learning . Adv Neural Inf Process Syst   2017 ; 30 :6467–6476.

Rannen A , Aljundi R , Blaschko MB , Tuytelaars T.   Encoder based lifelong learning . Proceedings of the IEEE International Conference on Computer Vision , p. 1320 – 1328 , 2017 .

Liu X , Masana M , Herranz L ,  et al.    Rotate your networks: better weight consolidation and less catastrophic forgetting . 2018 24th International Conference on Pattern Recognition (ICPR) , p. 2262 – 8 . IEEE , 2018 .

Rusu AA , Rabinowitz NC , Desjardins G , et al.    Progressive neural networks . arXiv preprint arXiv:1606.04671, 2016 .

Mallya A , Lazebnik S . PackNet: adding multiple tasks to a single network by iterative pruning . Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , p. 7765 – 73 , 2018 .

De Lange M , Aljundi R , Masana M , et al.    A continual learning survey: defying forgetting in classification tasks . IEEE Trans Pattern Anal Mach Intell   2021 ; 44 ( 7 ): 3366 – 85 .

Masana M , Liu X , Twardowski B , et al.    Class-incremental learning: survey and performance evaluation on image classification . IEEE Trans Pattern Anal Mach Intell   2022 ; 45 ( 5 ): 5513 – 33 .

Alquicira-Hernandez J , Sathe A , Ji HP , et al.    scPred: accurate supervised method for cell-type classification from single-cell RNA-seq data . Genome Biol   2019 ; 20 ( 1 ): 1 – 17 .

Cao Z-J , Wei L , Shen L , et al.    Searching large-scale scRNA-seq databases via unbiased cell embedding with cell blast . Nat Commun   2020 ; 11 ( 1 ): 3458 .

Yang F , Wang W , Wang F , et al.    scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data . Nat Mach Intell   2022 ; 4 ( 10 ): 852 – 66 .

Chen L , He Q , Zhai Y , Deng M . Single-cell RNA-seq data semi-supervised clustering and annotation via structural regularized domain adaptation . Bioinformatics   2021 ; 37 ( 6 ): 775 – 84 .

Bae S , Na KJ , Koh J , et al.    CellDART: cell type inference by domain adaptation of single-cell and spatial transcriptomic data . Nucleic Acids Res   2022 ; 50 ( 10 ): e57 – 7 .

Zhai Y , Chen L , Deng M . scGAD: a new task and end-to-end framework for generalized cell type annotation and discovery . Brief Bioinform   2023 ; 24 ( 2 ): bbad045 .

Zhai Y , Chen L , Deng M . Generalized cell type annotation and discovery for single-cell RNA-seq data . Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37 , p. 5402 – 10 , 2023 .

Zhai Y , Chen L, Deng M . Realistic cell type annotation and discovery for single-cell RNA-seq data . Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , p. 4967 – 74 , 2023 .

Eraslan G , Simon LM , Mircea M , et al.    Single-cell RNA-seq denoising using a deep count autoencoder . Nat Commun   2019 ; 10 .

He K , Fan H , Wu Y , XIe S, Girshick R.   Momentum contrast for unsupervised visual representation learning . Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , p. 9729 – 38 , 2020 .

Khosla P , Teterwak P , Wang C , et al.    Supervised contrastive learning . Adv Neural Inf Process Syst   2020 ; 33 : 18661 – 73 .

Hua T , Wang W , Xue Z , Ren S, Wang Y, Zhao H.   On feature decorrelation in self-supervised learning . Proceedings of the IEEE/CVF International Conference on Computer Vision , p. 9598 – 608 , 2021 .

Huang L , Yang D , Lang B and Deng J . Decorrelated batch normalization . Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , p. 791 – 800 , 2018 .

Cao J , Packer JS , Ramani V , et al.    Comprehensive single-cell transcriptional profiling of a multicellular organism . Science   2017 ; 357 ( 6352 ): 661 – 7 .

Schaum N , Karkanias J , Neff NF , et al.    Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris: the Tabula Muris Consortium . Nature   2018 ; 562 ( 7727 ): 367 .

Zeisel A , Hochgerner H , Lönnerberg P , et al.    Molecular architecture of the mouse nervous system . Cell   2018 ; 174 ( 4 ): 999 – 1014.e22 .

Cao J , O’Day DR , Pliner HA , et al.    A human cell atlas of fetal gene expression . Science   2020 ; 370 :808–850.

He J , Yan J , Wang J , et al.    Dissecting human embryonic skeletal stem cell ontogeny by single-cell transcriptomic and functional analyses . Cell Res   2021 ; 31 ( 7 ): 742 – 57 .

Madissoon E , Wilbrey-Clark A , Miragaia RJ , et al.    scRNA-seq assessment of the human lung, spleen, and esophagus tissue stability after cold preservation . Genome Biol   2020 ; 21 ( 1 ): 1 – 16 .

Stewart BJ , Ferdinand JR , Young MD , et al.    Spatiotemporal immune zonation of the human kidney . Science   2019 ; 365 ( 6460 ): 1461 – 6 .

Vento-Tormo R , Efremova M , Botting RA , et al.    Single-cell reconstruction of the early maternal–fetal interface in humans . Nature   2018 ; 563 ( 7731 ): 347 – 53 .

Jing X , Zhang A , Liu F , et al.    CIForm as a transformer-based model for cell-type annotation of large-scale single-cell RNA-seq data . Brief Bioinform   2023 ; bbad195 .

Van der Maaten L , Hinton G . Visualizing data using t-SNE . J Mach Learn Res   2008 ; 9 ( 11 ).

Supplementary data

Email alerts, citing articles via.

  • Recommend to your Library

Affiliations

  • Online ISSN 1477-4054
  • Copyright © 2024 Oxford University Press
  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Institutional account management
  • Rights and permissions
  • Get help with access
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

APRE: Annotation-Aware Prompt-Tuning for Relation Extraction

  • Open access
  • Published: 21 February 2024
  • Volume 56 , article number  62 , ( 2024 )

Cite this article

You have full access to this open access article

  • Chao Wei 1 , 2 ,
  • Yanping Chen 1 , 2 ,
  • Kai Wang 1 , 2 ,
  • Yongbin Qin 1 , 2 ,
  • Ruizhang Huang 1 , 2 &
  • Qinghua Zheng 3  

Prompt-tuning has been successfully applied to support classification tasks in natural language processing and has achieved promising performance. The main characteristic of prompt-tuning based classification is to verbalize class labels and predict masked tokens like a cloze-like task. It has the advantage to make use of knowledge in pre-trained language models (PLMs). Because prompt templates are manually designed, they are more prone to overfitting. Furthermore, traditional prompt templates are appended in the tail of an original sentence. They are far from some semantic units in a sentence. It is weak to decode semantic information of an input relevant to PLMs. To aggregate more semantic information from PLMs for masked token prediction, we propose an annotation-aware prompt-tuning model for relation extraction. In our method, entity type representations are used as entity annotations. They are implanted near the site of entities in a sentence for decoding semantic information of PLMs. It is effective to make full use of knowledge in PLMs for relation extraction. In the experiment section, our method is validated on the Chinese literature text and SemEval 2010 Task datasets and achieves 89.3% and 90.6% in terms of F1-score, respectively. It achieves the state-of-the-art performance on two public datasets. The result further demonstrates the effectiveness of our model to decode semantic information in PLMs.

Avoid common mistakes on your manuscript.

1 Introduction

Relation extraction (RE) is a fundamental task in information extraction, which identifies semantic relationships between two named entities within a sentence. It was widely adopted to construct Knowledge Graphs (KG) and support applications, such as information extraction, question answering, and construction of semantic networks. Extracting entity relations is key to understand semantic meaning of a sentence. Therefore, the task has received great attention in recurrent years [ 1 , 2 , 3 , 4 ]. Despite the great success that has been achieved in this field, it still remains a challenging task due to the reason that a sentence usually contains very little information, which leads to a serious sparse features.

Pre-trained Language Models (PLMs) are trained from external resources with unsupervised methods. PLMs usually consist of billions of parameters automatically learned from external resources, which encode rich knowledge of sentences that are valuable for relation extraction. They are effective to learn semantic information of tokens and the semantic dependencies of a sentence. The performance of many tasks in Natural Language Processing (NLP) has been substantially improved by the emergence of PLMs, such as BERT, GPT3 and etc. [ 5 , 6 ]. Supported by PLMs,relation extraction tasks also achieved state-of-the-art results in many benchmarks [ 7 , 8 ].

The methods to utilize PLMs can be roughly divided into two categories: fine-tuning and prompt tuning. In fine-tuning models, PLMs is used as a lookup table, which maps each token into a dense vector encoded with pre-trained semantic information. Then, a classifier is trained to make a classification based on abstract representations of a sentence. In the training process, PLMs are also tuned according to the classification objectives. In prompt tuning models, the classification is implemented as a cloze-like task, where class labels are verbalized and predicted as masked tokens. In prompt tuning models, prompt templates with slots are used to decode semantic information from PLMs. An example of fine-tuning and prompt tuning is shown in the middle of Fig.  1 , where the original sentence “The sergeant got out of the car.” contains two named entities “sergeant” and “car”.

figure 1

Examples of fine-tuning and prompt-tuning for Relation Extraction. The yellow rectangles in the figure are special tokens. (Color figure online)

The top of Fig.  1 is a fine-tuning approach, where the output of PLMs is directly fed into a classifier, which generates confidence scores for each relation type. In the middle of Fig.  1 , a prompt template “sergeant [MASK] car” is employed and concatenated with the original sentence. Then, they are fed into PLMs to predicate the masked token slots. The process to predict prompt template slots is the same as masked language model training. If it outputs “origin” verbalize class tokens, it indicates a “Entity-Origin” relation between “segeant” and “car”. Compared to fine-tuning models, prompt tuning has the advantage to make full use of knowledge in pre-trained language models (PLMs) and bridge the gap between pre-training objectives and fine-tuning objectives. Therefore, in recent years, prompt-tuning has been widely applied to support relation extraction [ 9 , 10 , 11 , 12 , 13 ].

Even great success has been achieved in prompt tuning models. Current RE models with prompt-tuning still confronts two significant challenges. First, because prompt templates are manually designed, it is easy to overfit the evaluation dataset. The migration between different domains is difficult. Second, previous prompt-tuning methods do not adequately utilize contextual features of a sentence. These models simply stitching prompts into the tail of sentences are ineffective to capture the contextual features of a sentence. It hinders the learning of the context of the entity in masked positions. For example, the type of “car” may be “destination” or “origin.” And the type of “car” is ambiguous according to the previous prompt-tuning methods. However, as shown in the bottom of Fig.  1 , we can intuitively imply from the semantics of “got out of the car” that the type of “car” is “origin”. In other words, the context contains many significant elements that help to understand relations among relation instances.

To aggregate more semantic information from PLMs for masked token prediction, we propose an annotation-aware prompt-tuning model for relation extraction (APRE). In our method, instead of manually designed prompt templates, entity annotations, implanted within a sentence, are adopted to decode semantic information of PLMs. Entity annotations are entity type representations, which contains semantic information about named entities. Instead of appending to the tail of a sentence by prompt templates, annotations are used as supplement features of named entities and implanted with named entities for semantic disambiguation. Because these annotations are near named entities in a sentence, it is effective to lean contextual features and semantic dependencies of the sentence. It also has the advantage to utilize potential knowledge of PLMs. We summarize our work contributions as follows:

We propose an Annotation-aware Prompt tuning for Relation Extraction that transforms prompts into annotation with contextual features. By introducing annotation prompts, the model can decode semantic information of PLMs.

Our proposed approach achieves state-of-the-art performance on two public datasets, SemEval (English) and CLTC (Chinese). Experimental results demonstrate the advancement and scalability of our method.

The rest of this paper is mainly divided into four different sections. Section  2 discusses the related works on RE. Section  3 details the annotation prompt. Section  4 shows the evaluation of the annotation prompt on Chinese and English datasets and presents the relevant experimental analysis. The conclusion is given in Sect.  5 .

2 Related Work

The task of RE is usually regarded as a classification problem, which makes a predication based on features about an entity pair of a relation instance. In recent years, PLMs such as GPT [ 6 ], BERT [ 5 ], and RoBERTa [ 14 ] have been widely adopted to support RE. These PLMs are trained by leveraging a large-scale unlabeled data, which provides rich semantic knowledge for the RE task [ 15 ]. In this section, we divided related work to utilize PLMs for RE into two categories: fine-tuning based models and prompt-tuning based models.

Fine-tuning based models are the traditional strategy to utilize PLMs. It has achieved great success, and the performance of RE has been improved considerably [ 8 , 16 , 17 , 18 , 19 , 20 ]. Since a relation is defined as a semantic relationship relevant to two named entities in a sentence, related works try to learn a structural sentence representation relevant to two named entities. For example, Wu and He [ 16 ] utilize a pre-trained architecture to locate target entities for enhancing sentence representations with encodings of entities. They propose an R-BERT model that could utilize information among PLMs and entities.

Because a sentence usually contains multiple entity pairs that share the same contextual features, it is also important to encode structural information of a relation instance. Chen et al. [ 18 ] introduced a neuralized feature engineering method that leverages manually designed features to improve the ability of neural networks for learn structural features. Zhao et al. [ 21 ] proposed a relation extraction method based on heterogeneous graph neural networks. They represent words and relations as nodes in a graph, which can iteratively obtain more appropriate node representations by message passing mechanisms. Zhou and Chen [ 20 ] utilize entity representations with TYPE MARKER to improve the baseline model performance. This approach virtually mitigates the influence of incorrectly defined labels.

Compared to general fine-tuning methods, knowledge-enhanced PLMs have significant performance improvements. Soares et al. [ 8 ] introduce extra knowledge via entity-linked text to construct agnostic task relation representations. They retrieve entity embeddings using entity links and update the context representation among words and entities. By building multiple knowledge bases of structured and human-curated knowledge, Peters et al. [ 17 ] have further enhances the knowledge representations of the model. The above methods of fine-tuning PLMs have proven to be successful on multiple datasets.

The prompt-tuning strategy is proposed in GPT-3 [ 6 ]. It has attracted the attention of researchers in various fields in recent years. This strategy leverages the prompt as context to alleviate the gap among tasks and PLMs by representing downstream tasks as pre-trained tasks. Recently, a series of studies [ 7 , 12 , 13 , 22 , 23 , 24 ] have demonstrated the effectiveness of PLMs. Because prompts have the ability to decode knowledge of PLMs, it achieved excellent performance on multiple NLP tasks. Ding et al. [ 22 ] aim to construct prompt templates and entity-oriented verbalizers. Then, they are utilized to explore entity types in prompt-tuning. Schick et al. [ 10 ] propose a method to avoid the laborious prompt design by studying the automatic exploration process of templates and answer words. Li and Liang [ 23 ] embed discrete template words as learnable continuous words, making the connection between template words closer.

During the process to design prompt templates, manual design is time-consuming and laborious. On the other hand, automated searches are resource-intensive. Hence, to support relation classification, Han et al. [ 12 ] construct prompts with multiple sub-prompts based on logical rules and proposed prompt tuning with rules (PTR). This method effectively reduces the labor costs caused by manual design and the waste of resources generated by automatic generation. Chen et al. [ 7 ] adopt a knowledge-aware approach with synergistic optimization in prompt tuning. They inject knowledge into the learnable continuous words, so as to the template words and answer words can acquire relevant knowledge during the prompt-tuning. It optimizes the representation of the prompt of the virtual template and answers words under the constraints of knowledge.

3 Methodology

Before introducing our method, we first give a formalized discussion about fine-tuning and prompt-tuning in relation extraction.

A relation instance is defined as a 3-triple \( I =\ \left\langle e_{s},e_{o}, r \right\rangle \) . Where \(e_{s}\) and \(e_{o}\) denote to two named entities, \( r \) denotes to a relation mention, which consists of a token sequence \( r \ = \ [t_1,t_2,\ldots ,t_n]\) . Named entity \(e\ =\ [t_i,\ldots ,t_j]\ (e\in \{e_s,e_o\})\) is a sub-sequence of \( r \) . Meanwhile, let \(\textbf{Y}\ =\ \{y_0,y_1,\ldots ,y_n\}\) be a set of relation categories, where \(y_0\) represents a negative relation type. \(\textbf{I}\ =\ \{I_1,I_2,\ldots ,I_m\}\) represent a relation instance set. Then, the task of RE can be expressed as a mapping process between \(\textbf{I}\) and \(\textbf{Y}\) .

Previous fine-tuning methods first utilize a pre-trained language model to map an input relation mention \( r= \{w_{\mathrm {\texttt {[CLS]}}}, w_1,w_2, \ldots w_s, \ldots ,w_o \ldots ,w_n,w_{\mathrm {\texttt {[SEP]}}}\}\) into abstract representation. The mapping has the ability to learn semantic features and contextual dependencies between tokens in a relation mention. The process is formalized as follows:

Generally, in fine-tuning methods, the output layer is a classifier composed of a multilayer perceptron layer and a softmax layer, which outputs the probability distribution of I on the relation label set \(\textbf{Y}\) :

where \(\mathrm {{\textbf {W}}}\) and \({\textbf {b}}\) are learnable parameters, \(\textrm{H}_{\mathrm {\texttt {[CLS]}}}\) is the output vector of \(\mathrm {\texttt {[CLS]}}\) , which can be used as the abstract representation of r . \(p(y\mid x)\) ( \(y\in \textbf{Y}\) ) represents the probability distribution under each category. Finally, the model utilizes the entire training set to minimize the cross-entropy loss to tune parameters.

On the other hand, prompt-tuning adopts a cloze-style framework. In this strategy, prompt templates with masked slots are designed and concatenated with a relation mention. It is fed into a pre-trained language model to predict verbalize label categories. It is effective to decode the potential knowledge of a pre-trained language model ( \(\mathcal {M}\) ). In this framework, constructing appropriate prompts is important. At current, prompt templates are often manually designed. The process to concatenate prompt template with a given relation mention I can be denoted as:

\(\mathcal {T}(\cdot )\) denotes to a process to concatenate an appropriate prompt template T with I . \(\oplus \) denotes to the concatenate operation. The output \(I_{prompt}\) is a token sequence composed of the original sentence I and a prompt template T . The prompt template T is a special token sentence containing mask tokens “ [MASK] ”. An example of prompt-tuning for relation extraction is shown in the middle of Fig.  1 , where a relation instance I are mapped to \(I_{prompt}\) = “ [CLS] I . sergeant [MASK] car [SEP] ”. The special token [MASK] corresponds a set of answer words in verbalized label set \(\mathcal {V}\) . Then, \(I_{prompt}\) is fed into a pre-trained language model \(\mathcal {M}\) to learn the hidden representation \(\mathrm {H_{\texttt {[MASK]}}}\) of token [MASK] . The pre-trained language model \(\mathcal {M}\) predicts masked token as an answer word \(\mathcal {V}_I\) with the highest score:

In prompt-tuning, the process to extract entity relations is formalized as:

where \(\mathrm {H_{\texttt {[MASK]}}}\) refers to the hidden layer vectors of token [MASK] , \(\mathcal {V}_I\) means the corresponding answer word of a relation instance I . As shown in Fig.  1 b, predictions for the relation categories Entity - Origin ( e 1,  e 2) and Entity - Destination ( e 1,  e 2) can be expressed as:

To summarize, the process to extract relations in prompt-tuning is the same as the process to train a language model. It bridges the gap between pre-training objectives and fine-tuning objectives. It has the advantage to make use of potential knowledge in a pre-trained language model.

figure 2

The methods flow of APRE. The yellow rectangles indicate special tokens [MASK] . In particular, the vanilla prompt also contains a [MASK] token. (Color figure online)

Our model, the Annotation-aware Prompt-tuning Model for Relation Extraction (APRE), aims to aggregate more semantic information from pre-trained language models for masked token prediction. Unlike manually designed prompt templates, our method implants entity annotations within a sentence to decode the semantic information of pre-trained language models, thereby improving the overall performance of our model. The architecture of our model to support annotation-aware prompt-tuning is given in Fig.  2 . This model is composed of three parts: annotation prompt construction, annotaion-aware module, and feature interaction mapping.

3.1 Annotation Construction

During the annotation construction phase, annotations are constructed for each entity. In previous studies, Zhou and Chen [ 20 ] leverage entity type information from prior knowledge as entity annotations to improve performance. Therefore, it is reasonable to assume that entities contain a wealth of knowledge of entity types. The pattern of entity annotation leverages the entity as the target and the context as the perceptual information, further mining the connections between them.

In our method, entity type tokens are adopted as entity annotations. They are implanted near named entities in the same sentence. Formally, given a relation instance \(I(e_s,e_o,r)\) , the constructor \( g (e_s)\) constructs entity annotations for each entity within the input. It is expressed as follows:

An example of the constructor \( g (e_s)\) is shown in the input part of Fig.  2 , “sergeant” is a named entity with entity type “soldier”. The entity type is implanted into the original sentence as “The sergeant (soldier) got out of the car”. Then, “soldier” serves as an annotation that reflects a connection to sergeant.

APRE can tune the PLMs and be aware of the context of [MASK] . We choose entity types based on prior knowledge as candidates \(\mathcal {V}_{e_{s}}\) for entity annotation prompt templates:

The structure of the annotation and the set of answer candidates for the corresponding prompt template for the objective entity \(e_o\) is similar to Eqs. ( 7 ) and ( 8 ).

We follow the vanilla prompts constructed by the previous prompt-tuning method [ 12 ] to restrict the relationship between entity pairs. The difference is that we have increased the verbalization of the vanilla prompt, making it closer to training text features and appropriately exploiting the MLM mechanism to mine potential connections between entity pairs. For instance, given an entity pair \((e_s,e_o)\) , we construct a vanilla prompt by constructor \(\textrm{g}(e_s,e_o)\) . Its corresponding answer template word \(W_{e_{s},e_{o}}\) can be expressed as:

For complex relationships of various entity pairs, a single prompt may not be enough to predict the target category. We adopt multiple annotation approach to construct prompts, leveraging the PLMs to learn the deeper semantic connections of the context between entity pairs. For the relation instance I , Table 1 shows the input of APRE after constructing annotation prompts.

3.2 Annotation Aware

In our model, a multi-head attention mechanism [ 25 ] is used in the annotation-aware module to obtain contextual semantic information of annotation prompts. Furthermore, we leverage annotation prompts to decrease the loss between the predictor and the answer word to optimize APRE.

Multi-head attention brings together different knowledge from multiple attention heads derived from different representation subspaces of Query ( Q ), Key ( K ), and Value ( V ). Through annotation prompts, multiple attention extracts characteristic information of a sentence from multiple dimensions. It is effective to prompt the acquisition of contextual semantics while paying attention to the special token [MASK] . The attention distribution \(\alpha _{\mathrm {\texttt {[MASK]}}}^{i}\) of ith [MASK] in relation instance I can be expressed as:

where \(q_{\mathrm {\texttt {[MASK]}}}^{i}\) denotes to the query of ith [MASK] . \(K^{T}\) represents the transpose of the matrix K and \(\sqrt{d_k}\) implies the scaling factor.

With ith single attention head, the attention \(head_i\) is represented as follow:

where \(Q_i=QW_i^Q\) , \(K_i=KW_i^K\) and \(V_i=VW_i^V\) . The projections \(W_i^Q\) , \(W_i^K\) and \(W_i^V\) are parameter matrices.

Subsequently, the output of the multi-head attention concatenate all heads and obtains the output through the dot product parameter matrice \(W_O\) :

Finally, the annotation aware module outputs the hidden layer vector through a fully connected feed-forward neural network.

After acquiring the output of the annotation-aware module, we need to predict the relation category based on representations of masked tokens [MASK] . Generally, prompt-tuning methods do not need an additional classifier. The training process is similar to the pre-training phase. However, in the case of prompt answer words, they are formally represented as the backbone of a sentence. Therefore, for a set of answer words, they have a non-negligible semantic connection. In order to further obtain the connection between them, when obtaining the prompt representation of the model output, we get the final semantic representation of the different position prompts through a feature interaction mapping, which will be discussed in following section.

3.3 Feature Interaction Mapping

To alleviate the prediction bias of PLMs on prompts, we leverage feature interaction mapping to obtain the final semantic representation of [MASK] in the prompts. In this module, we first concatenate three [MASK] token representations to construct the feature fusion vector \(\textrm{H}_{interact}\) :

where \(\mathrm {h_{\texttt {[MASK]}}^{i}}\) indicates the output of the ith special token [MASK] in the annotation aware module.

Then, we get the final representation \(\mathrm {H_{\texttt {[MASK]}}^{i}}\) of each special token [MASK] through three individual MLP:

Finally, we map the final output vectors of all special tokens [MASK] onto the corresponding relation category. Since annotation prompts are composed of multiple prompts, and each prompt is similar to a conditional function, our regular classification task is converted into the form of multi-condition functions. Meanwhile, as the multi-prompt template contains multiple [MASK] tokens, all the masked positions are applied to the prediction when we get the final output vectors. For example, the i th masked position corresponds to a set of answer words \(w \in \mathcal {V}\) . Given \(v \in w\) , we can calculate the probability distribution of the token v over w :

where v means the embedding of v in the pre-trained model and \(I_{prompt}\) refers to relation instance I with annotation prompts. For each relation instance, the probability distribution \(p(y\vert I)\) can be expressed as follows:

where n means the number of [MASK] token, w ( y ) indicates the answer words for the i th masked position of y . The answer words in annotation prompts are defined based on semantic knowledge of entities and relationships whose underlying spatial variables are associated with entities. Thus, we perform semantic optimization of the masked position through the semantic knowledge of entities. Furthermore, we mitigate the gap between PLMs and RE by calculating the cross-entropy loss of the prediction words \(p(y\vert I)\) and the answer words of relation category y :

where \(L_{\mathrm {\texttt {[MASK]}}}\) is the losses for [MASK] predictions. During the APRE training process, we enter the training data in batches. Ultimately, for each batch of data, the loss calculation and prediction are aimed at maximizing:

where \(\mathcal {X}\) represents the entire training collection. Masked positions in the prompt tuning can automatically sense entity information through the annotation-aware mechanism to learn the optimal representations of entity types.

4 Experiments

4.1 datasets and settings.

In this section, we conduct experiments on SemEval 2010 Task 8 (English) and analyse the results to demonstrate the effectiveness of our method. Meanwhile, our method is evaluated on the Chinese literature text corpus (Chinese) dataset to show the scalability of our proposed approach. The statistics of datasets as shown in Table 2 .

SemEval 2010 Task 8 (SemEval [ 26 ]) released on task 8 of the 2010 Semantic Evaluation Conference is a publicly available dataset in English. It contains 10,717 annotation examples divided into 8000 examples for training and 2717 examples for testing. Other than that, SemEval has a total of 18 relation types besides Other . In the dataset, all relations are directional. For example, Product-Producer(e1, e2) is different from Product-Producer(e2, e1).

Chinese literature text corpus (CLTC [ 27 ]) is a discourse-level corpus whose articles are made by manually labeling 837 Chinese essays. For facilitating comparison among different models, the dataset is divided into the training, validation, and test sets in advance. The corpus contains a total of 7 entity types and 9 relation types. Specially, there is no negative relation category in the CLTC dataset. In this experiment, our optimal performances are obtained on RoBERTa_WWM-LARGE (CLTC) and RoBERTa_LARGE (SemEval), respectively. We also experiment with the same PLMs to ensure the fairness of comparison. We follow the other hyper-parameter in the previous studies [ 7 , 12 , 28 ] and compare our model with related work: we optimize the model using the Adam optimizer with a learning rate of \(3e\!-\!5\) ;the value of warmup steps is 0.1; epochs and batch sizes are set to 20 and 16, respectively. Meanwhile, all of our experiments are performed on Nvidia A100 GPU. The F1 scores are used as the primary metric for evaluating the model. We select the checkpoint with the best validation performance to test during the tuning of the model. Most of other setup refers to the previous work [ 12 , 28 ]. The code to implement our model is available at: https://github.com/weichao21gzu/APRE .

4.2 Comparing with Other Models

In this section, our APRE model is compared to several related work. Based on the architecture of models, we roughly divide them into three strategies: Traditional neural networks, Fine-tuning methods, and Prompt-tuning methods. They are introduced as follows:

CR-CNN [ 29 ] considers only the text between the entity pair and uses it as input features for relation classification. CR-CNN achieves better performance without using any handcrafted features.

BRCNN [ 30 ] exploits dependencies on the shortest dependency path (SDP) by combining convolutional neural networks and long-short-term memory neural networks for RE. This approach leverages a bidirectional architecture to model the directional representation of relations.

SR-BRCNN [ 31 ] obtains the dependency relationship of entity pairs through building two bidirectional LSTMs on SDP. SR-BRCNN reduces the complexity of the overall model by learning relation representations by extracting the shortest dependency path, which facilitates the handling of text data with plenty of redundant information.

R-BERT [ 16 ] combines target entity information with contextualized word embeddings to perform relation extraction. This method utilizes a pre-trained architecture to transfer both sentence and entity information by identifying the target entities.

KnowBERT [ 17 ] retrieves entity embeddings by using entity links and updates the context representation between words and entities. This method enhances the knowledge representation of a model by building multiple knowledge bases of structured, human-curated knowledge, which are embedded into large models.

MTB [ 8 ] investigates ways to generate available representations of relationships directly from the original text. It constructs a task agnostic relation representation from the text of entity-linked based on extensions of Harris’ distributional hypothesis to relation.

RELA [ 32 ] RELA investigates the advantages of using sequence generation on RE, generating semantically equivalent synonyms for each relation name as the generation target, and exploring the impact of their textual semantics and correlations (word sequence patterns) on model performance. This method uses a generative model to complete the relation extraction task and provides an in-depth analysis of the Seq2Seq model’s ability to handle RE behavior.

KLG [ 33 ] explores whether the Top-k predicted set for a given sample contains useful information for predicting the correct label. The method effectively utilizes the Top-k predicted set to construct a label graph and examine the candidate labels in the set. Furthermore, the approach designs a dynamic k-selection mechanism that learns stronger and more discriminative representations for the relations.

SPOT [ 34 ] pretrains the model to learn representations of entities and relationships by using span and span-pair learning from the text during the pretraining phase. This approach represents the relations between entities by effectively encoding span-pair modules. In addition, this method utilizes a knowledge graph extracted from Wikipedia as an external knowledge base and applies it to the model’s pretraining.

PTR [ 12 ] constructs prompts with multiple sub-prompts based on logical rules and adapts prompt tuning with rules for relation classification. PTR encodes each class into prompt tuning through prior knowledge.

KnowPrompt [ 7 ] exploits a knowledge-aware approach with synergistic optimization in prompt tuning. Together, it optimizes the representation of the prompt of the virtual template and answers words under the constraints of knowledge.

BERT-CNN [ 4 ] implants structured entity indicators into the entity pair, which facilitates neural networks to encode syntax and learn semantic information. The entity indicators make the similar or identical internal symmetry of one sentence and entity pair more obvious.

EA-BERT [ 28 ] proposes a relation classification model based on the attention of entities for the Chinese literary text. It improves performance by filtering out redundant content and using the Attention Mechanism to extract critical information from the relation instances.

Our APRE model is compared with several typical methods for RE. It is mainly compared with other prompt tuning methods on SemEval. To verify the scalability of APRE on the Chinese dataset, we compare it with traditional neural networks and fine-tuning methods on CLTC. Since CR-CNN and BRCNN are the most efficient models based on traditional neural networks, we chose them as our baselines. The performance is shown in Tables  3 and 4 .

Traditional neural networks design complicated neural network architectures based on syntactic, lexical, and semantic features, which have achieved excellent performance effects. However, general neural network architectures have limitations in feature extraction. On the other hand, because English sentences usually have formal syntactic structures, traditional neural networks can benefit from syntactic knowledge of English sentences. However, Chinese sentences are not sensitive to sentence structures, it is difficult to use structured features in Chinese sentences. Therefore, the CLTC dataset has lower performance with traditional neural networks. The PLMs, which capture rich semantic knowledge from large-scale unlabeled data, exhibiting their powerful semantic representation. It achieves robust performance on two datasets. Its performance is superior to that of conventional methods in many downstream tasks.

Fine-tuning models leverage the plentiful semantic knowledge of PLMs, which can be used as a feature extractor that effortlessly obtain powerful representations of words or sentences. Further, external knowledge is encoded into PLMs, so that it can learn the representation of task-specific features. Fine-tuning can achieve better performance than conventional methods. In particular, the effect is evident in the CLTC dataset. BERT-CNN improves performance by 11.2% compared to SR-BRCNN. On the other hand, there is a mass of redundant information in Chinese literature. EA-BERT filters redundant content and utilizes an attention mechanism to extract critical information from relation instances. Its performance is approximately 9.1% better than BERT-CNN. Unlike simple fine-tuning methods, KnowBERT and MTB each employ different fine-tuning strategies and improve model performance. KnowBERT leverages knowledge base for data augmentation during the pre-training phase, enhancing the distributed representation of PLMs. Its performance is about 5.0% better than CR-CNN on SemEval. Besides, MTB enables the model to learn characteristic representations during the pre-training phase by injecting task-specific knowledge into the model parameters. The model is then fine-tuned without extra knowledge. The above results show that fine-tuning has more superiority over conventional methods. SPOT pre-trains the model to learn representations of entities and relationships by using span and span-pair learning from the text during the pretraining phase. This approach represents the relationships between entities by effectively encoding span-pair modules. In addition, this method utilizes a knowledge graph extracted from Wikipedia as an external knowledge base and applies it to the model’s pre-training. To sum up, PLMs serve a wide range of downstream tasks so that they are not highly pertinence. In particular, when the sample size of the fine-tuned data is small, the effect of fine-tuning is negligible.

Prompt-tuning models bridge the gap between specific tasks and PLMs. It alleviates the inability to fine-tune the model when there are fewer samples. PTR constructs prompts with multiple sub-prompts based on logical rules. Compared to the fine-tuning methods on SemEval, it has a slight performance improvement. KnowPrompt exploits a knowledge-aware approach with synergistic optimization, which has a 0.2% performance improvement over PTR on. In contrast, APRE is more pronounced in Chinese text than in fine-tuning methods. Due to the difference in pre-trained language models, their MLM are also different. Prompt tuning relies on MLM capabilities. Nevertheless, APRE performs better than current fine-tuning methods under the same conditions. We reproduced the PTR method on the CLTC dataset. APRE has improved performance by approximately 1.3% compared to this.

4.3 Performance on Few-Shot Learning

PLMs contain a large number pretrained parameters. They encode rich semantic knowledge, which is beneficial for few-shot learning. In this experiment, we evaluates the feasibility of our model for few-shot learning. We randomly collect K samples (K-shots, \(K \in [8, 16, 32]\) ) from each relation class in the training dataset. These samples are used to tune the PLMs. Then, the PLMs are evaluated on the entire test dataset. A fine-tuning and a prompt-tuning model (R-BERT and PTR) are conducted as baselines for comparison. The experimental results are listed in Table  5 .

The experimental results indicate that prompt-tuning is more suitable for few-shot learning than the compared to the fine-tuning methods. Because in fine-tuning models, several neural layers are stacked for extracting abstract features from the output of PLMs. It is not effective to optimize a fine-tunning model with very few samples. Therefore, the fine-tuning model achieves the lowest performance on all few-shot learning setting. On the other hand, the pre-training objective and fine-tuning objective are same in the prompt tuning. It is effective to make use of task-relevant features in PLMs. Comparing our model with the PTR model, our model steadly improves the performance in prompt-tuning. The result indicates that, benefited from the entity type representations, our annotation-aware prompt-tuning is more suitable for few-shot learning than the PTR methods.

As the number of samples are increased, the performance is steadily improved. The few-shot learning demonstrates impressive performance. It achieves competitive results at K = 32. Compared the performance between 16-shot and 32-shot, the performance of the R-BERT model has a large improvement, the result indicates that fine-tuning methods heavily depend on the amount of training data. The prompt tuning methods can quickly adjust PLMs with few training instances.

We also give the influence of training epochs. It is shown in Fig.  3 .

figure 3

Influence of training epochs on few-shot learning using the SemEval dataset

It is shown in Fig.  3 , our model exhibits faster convergence speed and higher performance. It also displayed stable performance during training, which is consistent with the results in Table 5 . Compared to the R-BERT model, the prompt-tuning method still also has robust performance. Experimental results show that annotated prompts can better stimulate the potential of pre-trained language models than general prompts, thereby proving the feasibility of the proposed method.

4.4 Analysis

In Sect.  4.4.1 , we design different ablation strategies to conduct ablation experiments on entity annotation and vanilla prompts to demonstrate the validity of annotations. In Sect.  4.4.2 , we show the evaluation performance of each relationship category in the Sem dataset. The error analysis is given in Sect.  4.4.3 . At the same time, we provide visual analysis in 4.4.4 .

4.4.1 Ablation

Entity annotation means, after learning the contextual information, we only calculate the probability distribution of the [MASK] in the entity type candidates to map the relation category. The vanilla prompt indicates that we rely solely on predicting the connections between entities to map the corresponding relation. As shown in Table 6 , by screening for validity for each module of APRE, we can discover that performance improves with each additional module.

Due to the different types of entity pair for predefined relationship categories in the SemEval dataset and their directionality, single Entity annotation prompts are already capable of making precise predictions. Nonetheless, there are possibilities in the dataset where the entity types are nearly identical, but the corresponding relationships are diverse. Accordingly, the predicate relationship between entity pairs is particularly significant. From the above experimental results, It can be concluded that adding Vanilla prompt to Entity annotations improves precision by 1.13% and the overall F1 score increases by 0.73%. Since in the absence of the negative relation category, F1-micro score is equal to the scores of precision and recall.

4.4.2 Category Analysis

To further prove that annotations in APRE can be aware of contextual information and to verify its effectiveness and stability of APRE, we research on the performance of each category in each module of APRE. The result is shown in Table 7 .

Table 7 shows the performance of each category in different ablation strategies. Overall, the APRE is superior to other strategies in most categories. In particular, the vanilla prompt is unstable, with it excelling on Entity - Destination and Other and poorer on Message - Topic and Memory - Collection . In addition, the entity annotation is already equivalent to PTR on performance. On this basis, APRE alleviates the instability of the vanilla prompt and improves the performance of the entity annotation. For the learning of negative samples, APRE inherits the advantages of the vanilla prompt and improves the anti-interference resistance of the model during training.

4.4.3 Error Analysis

To verify the exactness of annotation awareness, during the evaluation phase, we compare the predictions of our method, PTR, and FINE-TUNING with several typical instances. The result is shown in Table 8 .

As shown in Table 8 , all three methods can judge the correct result in the first instance. For the second and third examples, FINE-TUNING is incorrectly labeled Cause - Effect ( e 2,  e 1) and Instrument - Agency ( e 2,  e 1), respectively. PTR mistakenly predicts the third example is Entity - Destination ( e 1,  e 2). In contrast, APRE can leverage annotations to be aware of entities and contextual information, which is beneficial to labeling the relationship of the entity pair.

4.4.4 Visual Analysis

To further explore the ability of [MASK] in APRE to capture global and local semantics, we utilize BertViz [ 35 ] to visualize transformations from the attention of neurons. The result is shown in Fig.  4 .

figure 4

View of neurons for APRE and vanilla prompt. Attention scores are determined based on line weight. The three plots from left to right represent views of neurons at different layers

As shown in Fig.  4 , most neurons pay more attention to their previous or latter neurons in the first layer (left). As the neural network layer deepens, [MASK] gradually captures local semantic information. The attention of each neuron begins to shift to [MASK] (middle). From the Figure (right), each [MASK] ultimately captures the attention of the majority of neurons, thereby enhancing their semantic representations. To elucidate, the attention of sergeant lies in its previous word while also focusing on the adjacent [MASK] . The above proves that entity annotations can effectively leverage entity information to rich the representation of [MASK] . Similarly, entities and [MASK] pay attention to each other in vanilla prompt.

5 Conclusion

In this paper, we propose an APRE model to encode entity annotation features for relation extraction. It is an annotation awareness prompt tuning model, which adopts entity annotation prompts to utilize semantic information of PLMs. Experimental results demonstrate that our approach achieves state-of-art performances, significantly outperforming current methods. Compared with other prompt tuning methods, APRE improves the performance on the SemEval dataset. We leverage annotation prompt tuning on the Chinese CLTC dataset, which contains a large amount of redundant data and is insensitive to structure. We have also achieved the state-of-the-art performance in the Chinese CLTC dataset. Furthermore, we offer a localization prompt design strategy that can be employed for prompt-based models. It also has the potential to be extended for other information extraction task. In further work, we will utilize APRE for cue calibration in semi-supervised or unsupervised relationship extraction. Finally, we’ll focus on automating relation extraction through annotation-aware prompt tuning.

Data Availability

The source code and preprocessed datasets utilized in this study are publicly accessible on Github: https://github.com/Weichao21gzu/APRE .

Zhang N, Deng S, Sun Z, Chen X, Zhang W, Chen H (2018) Attention-based capsule networks with dynamic routing for relation extraction. In: Proceedings of the 2018 conference on empirical methods in natural language processing. Association for Computational Linguistics, Brussels, Belgium, pp 986–992. https://doi.org/10.18653/v1/D18-1120 ; https://aclanthology.org/D18-1120

Chen Y, Wang K, Yang W, Qing Y, Huang R, Chen P (2020) A multi-channel deep neural network for relation extraction. IEEE Access 8:13195–13203. https://doi.org/10.1109/ACCESS.2020.2966303

Article   Google Scholar  

Schick T, Schütze H (2021) Exploiting cloze-questions for few-shot text classification and natural language inference. In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume. Association for Computational Linguistics, Online, pp 255–269. https://doi.org/10.18653/v1/2021.eacl-main.20 ; https://aclanthology.org/2021.eacl-main.20

Qin Y, Yang W, Wang K, Huang R, Tian F, Ao S, Chen Y (2021) Entity relation extraction based on entity indicators. Symmetry. https://doi.org/10.3390/sym13040539

Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp 4171–4186. https://doi.org/10.18653/v1/N19-1423 ; https://aclanthology.org/N19-1423

Radford A, Narasimhan K (2018) Improving language understanding by generative pre-training

Chen X, Zhang N, Xie X, Deng S, Yao Y, Tan C, Huang F, Si L, Chen H (2022) Knowprompt: knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In: Proceedings of the ACM web conference 2022. WWW ’22. Association for Computing Machinery, New York, NY, USA, pp 2778–2788. https://doi.org/10.1145/3485447.3511998 ; https://doi.org/10.1145/3485447.3511998

Baldini Soares L, FitzGerald N, Ling J, Kwiatkowski T (2019) Matching the blanks: distributional similarity for relation learning. In: Proceedings of the 57th annual meeting of the association for computational linguistics. Association for Computational Linguistics, Florence, Italy, pp 2895–2905. https://doi.org/10.18653/v1/P19-1279 ; https://aclanthology.org/P19-1279

Schick T, Schütze H (2021) It’s not just size that matters: small language models are also few-shot learners. In: Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies. Association for Computational Linguistics, Online, pp 2339–2352. https://doi.org/10.18653/v1/2021.naacl-main.185 ; https://aclanthology.org/2021.naacl-main.185

Schick T, Schmid H, Schütze H (2020) Automatically identifying words that can serve as labels for few-shot text classification. In: Proceedings of the 28th international conference on computational linguistics. International Committee on Computational Linguistics, Barcelona, Spain, pp 5569–5578 (Online). https://doi.org/10.18653/v1/2020.coling-main.488 ; https://aclanthology.org/2020.coling-main.488

Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language models are few-shot learners. In: Proceedings of the 34th international conference on neural information processing systems. NIPS’20. Curran Associates Inc., Red Hook, NY, USA. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

Han X, Zhao W, Ding N, Liu Z, Sun M (2022) Ptr: prompt tuning with rules for text classification. AI Open 3:182–192. https://doi.org/10.1016/j.aiopen.2022.11.003

Lester B, Al-Rfou R, Constant N (2021) The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 conference on empirical methods in natural language processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, pp 3045–3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 ; https://aclanthology.org/2021.emnlp-main.243

Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized BERT pretraining approach. arXiv:1907.11692

Petroni F, Rocktäschel T, Riedel S, Lewis P, Bakhtin A, Wu Y, Miller A (2019) Language models as knowledge bases? In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, pp 2463–2473. https://doi.org/10.18653/v1/D19-1250 ; https://aclanthology.org/D19-1250

Wu S, He Y (2019) Enriching pre-trained language model with entity information for relation classification. In: Proceedings of the 28th ACM international conference on information and knowledge management. CIKM ’19. Association for Computing Machinery, New York, NY, USA, pp 2361–2364. https://doi.org/10.1145/3357384.3358119

Peters ME, Neumann M, Logan R, Schwartz R, Joshi V, Singh S, Smith NA (2019) Knowledge enhanced contextual word representations. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, pp 43–54. https://doi.org/10.18653/v1/D19-1005 ; https://aclanthology.org/D19-1005

Chen Y, Yang W, Wang K, Qin Y, Huang R, Zheng Q (2021) A neuralized feature engineering method for entity relation extraction. Neural Netw 141:249–260. https://doi.org/10.1016/j.neunet.2021.04.010

Article   PubMed   Google Scholar  

Lyu S, Chen H (2021) Relation classification with entity type restriction. In: Findings of the association for computational linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online, pp 390–395. https://doi.org/10.18653/v1/2021.findings-acl.34 ; https://aclanthology.org/2021.findings-acl.34

Zhou W, Chen M (2022) An improved baseline for sentence-level relation extraction. In: Proceedings of the 2nd conference of the Asia-Pacific chapter of the association for computational linguistics and the 12th international joint conference on natural language processing (volume 2: short papers). Association for Computational Linguistics, Online only, pp 161–168. https://aclanthology.org/2022.aacl-short.21

Zhao K, Xu H, Cheng Y, Li X, Gao K (2021) Representation iterative fusion based on heterogeneous graph neural network for joint entity and relation extraction. Knowl-Based Syst 219:106888. https://doi.org/10.1016/j.knosys.2021.106888

Ding N, Chen Y, Han X, Xu G, Wang X, Xie P, Zheng H, Liu Z, Li J, Kim H-G (2022) Prompt-learning for fine-grained entity typing. In: Findings of the association for computational linguistics: EMNLP 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, pp 6888–6901. https://aclanthology.org/2022.findings-emnlp.512

Li XL, Liang P (2021) Prefix-tuning: optimizing continuous prompts for generation. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers). Association for Computational Linguistics, Online, pp 4582–4597. https://doi.org/10.18653/v1/2021.acl-long.353 ; https://aclanthology.org/2021.acl-long.353

Wang K, Chen Y, Wen K, Wei C, Dong B, Zheng Q, Qin Y (2023) Cue prompt adapting model for relation extraction. Connect Sci 35(1):2161478. https://doi.org/10.1080/09540091.2022.2161478

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17. Curran Associates Inc., Red Hook, NY, USA, pp 6000–6010

Hendrickx I, Kim SN, Kozareva Z, Nakov P, Ó Séaghdha D, Padó S, Pennacchiotti M, Romano L, Szpakowicz S (2010) SemEval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. In: Proceedings of the 5th international workshop on semantic evaluation. Association for Computational Linguistics, Uppsala, Sweden, pp 33–38. https://aclanthology.org/S10-1006

Xu J, Wen J, Sun X, Su Q (2017) A discourse-level named entity recognition and relation extraction dataset for Chinese literature text, vol. abs/1711.07010. arxiv:1711.07010

Xie W (2021) A entity attention-based model for entity relation classification for Chinese literature text. In: 2021 IEEE 4th advanced information management, communicates, electronic and automation control conference (IMCEC), vol 4, pp 1104–1108. https://doi.org/10.1109/IMCEC51613.2021.9482227

Dos Santos C, Xiang B, Zhou B (2015) Classifying relations by ranking with convolutional neural networks. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing, vol 1. https://doi.org/10.3115/v1/P15-1061

Cai R, Zhang X, Wang H (2016) Bidirectional recurrent convolutional neural network for relation classification. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics, Berlin, Germany, pp 756–765. https://doi.org/10.18653/v1/P16-1072 ; https://aclanthology.org/P16-1072

Wen J, Sun X, Ren X, Su Q (2018) Structure regularized neural network for entity relation classification for Chinese literature text. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 2 (short papers). Association for Computational Linguistics, New Orleans, Louisiana, pp 365–370. https://doi.org/10.18653/v1/N18-2059 ; https://aclanthology.org/N18-2059

Li B, Yu D, Ye W, Zhang J, Zhang S (2022) Sequence generation with label augmentation for relation extraction. arXiv e-prints, pp. 2212–14266 https://doi.org/10.48550/arXiv.2212.14266 ; arXiv:2212.14266

Li B, Ye W, Zhang J, Zhang S (2022) Reviewing labels: label graph network with top-k prediction set for relation extraction. arXiv:2212.14270

Li J, Katsis Y, Baldwin T, Kim H-C, Bartko A, McAuley J, Hsu C-N (2022) Spot: knowledge-enhanced language representations for information extraction. In: Proceedings of the 31st ACM international conference on information and knowledge management. CIKM ’22. Association for Computing Machinery, New York, NY, USA, pp 1124–1134. https://doi.org/10.1145/3511808.3557459 ; https://doi.org/10.1145/3511808.3557459

Vig J (2019) A multiscale visualization of attention in the transformer model. In: Proceedings of the 57th annual meeting of the association for computational linguistics: system demonstrations. Association for Computational Linguistics, Florence, Italy, pp 37–42. https://doi.org/10.18653/v1/P19-3007 ; https://aclanthology.org/P19-3007

Download references

Acknowledgements

This work is supported by the funds of the National Natural Science Foundation of China (No. 62166007, No. 62066007, No. 62066008), the funds of the Guizhou Provincial Science and Technology Projects (No. ZK[2022]027, No. ZK[2022]227) and the fund of Servyou Group (No. 2023610002000756). Thanks to the editors and anonymous reviewers for their valuable suggestions and comments that made our final version of the paper more perfect.

This work is supported by National Natural Science Foundation of China under Grant No. 62166007.

Author information

Authors and affiliations.

State Key Laboratory of Public Big Data, Guizhou University, Huaxi, 550025, Guiyang, China

Chao Wei, Yanping Chen, Kai Wang, Yongbin Qin & Ruizhang Huang

College of Computer Science and Technology, Guizhou University, Huaxi, 550025, Guiyang, China

Xian’an Jiaotong University, Xi’an, 710049, China

Qinghua Zheng

You can also search for this author in PubMed   Google Scholar

Contributions

CW was mainly responsible for the investigation, methodology, experiments, preparing figures 1–4, visual analysis and writing the main manuscript text. KW was responsible for formal analysis, and experimental verification. YQ supervised the research project and collected research resources. RH was responsible for supervising research projects and reviewing the manuscript. QZ contributed experimental guidance and visual analysis. YC was responsible for the conceptualization, guidance of research topics, methodology, review, and revision.

Corresponding author

Correspondence to Yanping Chen .

Ethics declarations

Conflict of interest.

I hereby declare that the authors have no competing interests as defined by Springer, nor do they have any other interests that could be perceived as influencing the results and/or discussions presented in this paper.

Ethical Approval

Ethical approval is not required for this work.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Wei, C., Chen, Y., Wang, K. et al. APRE: Annotation-Aware Prompt-Tuning for Relation Extraction. Neural Process Lett 56 , 62 (2024). https://doi.org/10.1007/s11063-024-11437-y

Download citation

Accepted : 30 October 2023

Published : 21 February 2024

DOI : https://doi.org/10.1007/s11063-024-11437-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Relation extraction
  • Prompt tuning
  • Semantic information
  • Find a journal
  • Publish with us
  • Track your research

Jaspersoft Community

  • Remember me Not recommended on shared computers

Forgot your password?

  • Join the Community
  • Tips & Tricks

Extending OpenTelemetry feature in JasperReports Server application by Adding Custom Annotations

By Henna Katy

  • Tuesday at 07:37 AM

Annotations Overview

The span names and properties can be modified for better visibility on Jaeger. You can create an annotation by adding @WithSpan to the method. This creates a span for a method for which it is annotated with. These annotated method names appear under the Operations field in Jaeger UI. The following figure shows the @WithSpan annotation for one of the methods involved in the process of Scheduling.

image.thumb.png.3c73e162dc354886f532c181d11f5b5d.png

Adding Annotations

Use cases for Adding Annotation

You can add annotations in the method of the code to investigate the performance issues of JasperReports Server application during execution of the report. The performance of an application can be slow due to various reasons, for example, if a report is taking longer than usual to execute, in such cases adding annotations helps to break down the request time by detecting slow queries and capturing the exact places (spans) and the time spent in each major code area. 

For more information about various scenarios where adding annotations helped to improve the performance of JasperReports Server application, see  https://community.jaspersoft.com/knowledgebase/how-to/how-to-investigate-the-performance-issues-of-scheduler-and-input-controls-in-the-jasperreports-server-r4635/

Adding @WithSpan and Creating Custom Spans

 adding @withspan.

1. Add the OpenTelemetry library in the following pom.xml of the corresponding java module in which the class resides.

2. Import import io.opentelemetry.extension.annotations.WithSpan;

3. After importing add @WithSpan to the required method.

Creating Custom Spans

1. Add the following dependency in the pom.xml for creating a span using the Span class:

2. Import import io.opentelemetry.api.trace.Span;

Methods for Adding Annotations

The following are the main methods for adding annotations in the code. You need to set the following span names and properties for instrumenting the code:

  • span = Span.current()
  • span.updateName()
  • span.setAttribute() 

For more information, refer to OpenTelemetry javadoc  https://javadoc.io/doc/io.opentelemetry/opentelemetry-api-trace/latest/io/opentelemetry/api/trace/Span.html .

Like

Table of contents

User feedback, recommended comments.

There are no comments to display.

Guest

  • Knowledge Base
  • Documentation
  • All Activity
  • Create New...

© 2024 Cloud Software Group , Inc. All rights reserved.

  • Terms of Use
  • Privacy Policy
  • Cookie preferences
  • Jaspersoft commercial edition
  • Jaspersoft community edition
  • Download Jaspersoft
  • Jaspersoft Cloud BI

IMAGES

  1. How to Annotate

    an annotation method

  2. How To Annotate Example

    an annotation method

  3. how to write an annotation in lesson plan

    an annotation method

  4. Simple Guide to Annotation

    an annotation method

  5. Teaching literature, How to annotate text, Middle school reading

    an annotation method

  6. Mastering the Art of Annotation: Tips and Techniques for Effective Note-Taking

    an annotation method

VIDEO

  1. 09 annotate

  2. 131 Annotation Lab

  3. 6- Screening by reading Title and Abstract

  4. Lec29 a 2018 02 19 hibernate more

  5. My new annotation method 🤓 #christiancontent #booknerd #biblestudy #christiancommunity #annotation

  6. Markup SVG region annotation method

COMMENTS

  1. An Introduction to Annotations and Annotation Processing in Java

    An annotation is a construct associated with Java source code elements such as classes, methods, and variables. Annotations provide information to a program at compile time or at runtime based on which the program can take further action.

  2. Creating a Custom Annotation in Java

    Java annotations are a mechanism for adding metadata information to our source code. They're a powerful part of Java that was added in JDK5. Annotations offer an alternative to the use of XML descriptors and marker interfaces.

  3. Annotations in Java

    Annotations are used to provide supplemental information about a program. Annotations start with ' @ '. Annotations do not change the action of a compiled program. Annotations help to associate metadata (information) to the program elements i.e. instance variables, constructors, methods, classes, etc.

  4. Annotations Basics (The Java™ Tutorials

    The annotation type can be one of the types that are defined in the java.lang or java.lang.annotation packages of the Java SE API. In the previous examples, Override and SuppressWarnings are predefined Java annotations. It is also possible to define your own annotation type.

  5. Annotations

    The annotation type can be one of the types that are defined in the java.lang or java.lang.annotation packages of the Java SE API. In the previous examples, Override and SuppressWarnings are predefined Java annotations. It is also possible to define your own annotation type.

  6. Java Annotations

    Some important points about java annotations are: Annotation methods can't have parameters. Annotation methods return types are limited to primitives, String, Enums, Annotation or array of these. Java Annotation methods can have default values. Annotations can have meta annotations attached to them. Meta annotations are used to provide ...

  7. Java annotation

    Java annotation. In the Java computer programming language, an annotation is a form of syntactic metadata that can be added to Java source code. [1] Classes, methods, variables, parameters and Java packages may be annotated. Like Javadoc tags, Java annotations can be read from source files. Unlike Javadoc tags, Java annotations can also be ...

  8. Complete Java Annotations Tutorial

    An annotation is a kind of metadata in Java that can be applied to various elements of Java source code so that later some tool, debugger, or application program can take advantage of these annotations; and help analyze the program in a positive and constructive way. Just to mention, we can annotate classes, methods, variables, parameters, and packages in Java OR in one word almost everything.

  9. A Guide to Java Annotations with Examples

    Java annotations are one of the main ease-of-development features introduced in JDK 5. Annotations are like meta-tags that you can add to your code and apply to package declarations, type declarations, constructors, methods, fields, parameters and variables.

  10. Java Annotations Explained

    Annotation[] getDeclaredAnnotations()— Returns all annotations on the method. Annotation[][] getParameterAnnotations()— Returns a two-dimensional array, containing the parameter annotations, in declaration order. Use Cases. An excellent use case is Serialization. With annotations, a lot of additional metadata can be provided on how our data ...

  11. Java Annotation Essentials Guide

    Java annotations are one of the cornerstone features of modern Java development, offering a powerful way to describe and control the behavior of code elements. Introduced in Java 5 through JSR...

  12. Java Annotations (With Examples)

    Any declaration can be marked with annotation by placing it above that declaration. As of Java 8, annotations can also be placed before a type. 1. Above declarations. As mentioned above, Java annotations can be placed above class, method, interface, field, and other program element declarations. Example 2: @SuppressWarnings Annotation Example

  13. A Closer Look at Annotations in Java

    Annotations provide a structured way to document code. For example, the @Override annotation indicates that a method is meant to override a method in a superclass, improving code readability. @Override public void myMethod() {// Implementation here} Compiler Instructions. Annotations can provide instructions to the Java compiler.

  14. Java Annotations

    Java Annotation is a tag that represents the metadata i.e. attached with class, interface, methods or fields to indicate some additional information which can be used by java compiler and JVM. Annotations in Java are used to provide additional information, so it is an alternative option for XML and Java marker interfaces.

  15. Annotations Best Practices

    In that case, best practice relies on an implementation detail of Python 3.9 and before: if a class has annotations defined, they are stored in the class's __dict__ dictionary. Since the class may or may not have annotations defined, best practice is to call the get method on the class dict.

  16. Research Guides: Reading and Study Strategies: Annotating a Text

    Annotating is any action that deliberately interacts with a text to enhance the reader's understanding of, recall of, and reaction to the text. Sometimes called "close reading," annotating usually involves highlighting or underlining key pieces of text and making notes in the margins of the text.

  17. Function Annotations in Python

    It has a special 'help()' method that provides an interactive shell to get help on any keyword, method, class or module. 'help()' can be used to access the function annotations. The image below shows the function annotations in the above Fibonacci series code. The module name is 'fib.py'. 3.

  18. Annotating Texts

    A systematic summary of the text that you create within the document A key tool for close reading that helps you uncover patterns, notice important words, and identify main points An active learning strategy that improves comprehension and retention of information Why annotate? Isolate and organize important material Identify key concepts

  19. Annotate

    To annotate is to make notes on or mark up at text with one's thoughts, questions, or realizations while reading. The term annotation refers to the actual notes one has written during the process...

  20. reflection

    public static List<Method> getMethodsAnnotatedWith (final Class<?> type, final Class<? extends Annotation> annotation) { final List<Method> methods = new ArrayList<Method> (); Class<?> klass = type; while (klass != Object.class) { // need to traverse a type hierarchy in order to process methods from super types // iterate though the list of ...

  21. How to Annotate Texts

    How do you prepare? The resources linked in this section list strategies and techniques you can use to start annotating. What is Annotating? (Charleston County School District) This resource gives an overview of annotation styles, including useful shorthands and symbols.

  22. An event-based automatic annotation method for datasets of

    In this method, We design an event-based set to replace the knowledge set, which reduces the cost for reconstructing the knowledge set and offers considerably higher generalization ability. Moreover, we design an event alignment label-annotation mechanism and a scoring mechanism to reduce the possibility of inaccurate and incomplete annotation.

  23. What Is an Annotated Bibliography?

    Published on March 9, 2021 by Jack Caulfield . Revised on August 23, 2022. An annotated bibliography is a list of source references that includes a short descriptive text (an annotation) for each source. It may be assigned as part of the research process for a paper, or as an individual assignment to gather and read relevant sources on a topic.

  24. A-SOiD, an active-learning platform for expert-guided, data-efficient

    Nature Methods - A-SOiD is a computational platform for behavioral annotation whose training includes elements of supervised and unsupervised learning. The approach is demonstrated on mouse ...

  25. scEVOLVE: cell-type incremental annotation without forgetting for

    Recently, the scientific community has seen a surge in the development of automatic annotation methods aimed at this task. However, these methods generally operate at a steady-state total cell-type capacity, significantly restricting the cell annotation systems'capacity for continuous knowledge acquisition. Furthermore, creating a unified scRNA ...

  26. java

    Modified today. Viewed 4 times. 0. I want to access a method in an interface in my test class method in compile time just by using a custom annotation. I dont want to implement the interface by test class. Below are the examples of interface, custom annotation and Test class. public interface A { default Student prepareRequest () { return ...

  27. APRE: Annotation-Aware Prompt-Tuning for Relation Extraction

    Unlike manually designed prompt templates, our method implants entity annotations within a sentence to decode the semantic information of pre-trained language models, thereby improving the overall performance of our model. The architecture of our model to support annotation-aware prompt-tuning is given in Fig. 2. This model is composed of three ...

  28. Extending OpenTelemetry feature in JasperReports Server application by

    Annotations Overview The span names and properties can be modified for better visibility on Jaeger. You can create an annotation by adding @WithSpan to the method. This creates a span for a method for which it is annotated with. These annotated method names appear under the Operations field in Jaeger UI.