Add documentation

docs/classes/adapter_model_interface.rst

@@ -0,0 +1,8 @@
+Adapter Model Interface
+=======================
+
+.. autoclass:: adapters.AdapterModelInterface
+    :members:
+
+.. autoclass:: adapters.AdapterMethod
+    :members:

docs/contributing/adding_adapters_to_a_model.md

@@ -1,4 +1,11 @@
 # Adding Adapters to a Model
+
+```{eval-rst}
+.. important::
+    For most use cases, it can be much easier support a new model architecture via the new adapter plugin interface.
+    Check out `Custom Models <../plugin_interface.html>`_ for more.
+```
+
 This document gives an overview of how new model architectures of Hugging Face Transformers can be supported by `adapters`.
 Before delving into implementation details, you should familiarize yourself with the main design philosophies of `adapters`:
 

docs/index.rst

@@ -52,7 +52,6 @@ Currently, we support the PyTorch versions of all models as listed on the `Model
    merging_adapters
    prediction_heads
    embeddings
-   extending
 
 .. toctree::
    :maxdepth: 2
@@ -66,6 +65,7 @@ Currently, we support the PyTorch versions of all models as listed on the `Model
    :caption: Supported Models
 
    model_overview
+   plugin_interface
    classes/models/albert
    classes/models/auto
    classes/models/bart
@@ -99,6 +99,7 @@ Currently, we support the PyTorch versions of all models as listed on the `Model
    classes/adapter_config
    classes/model_adapters_config
    classes/adapter_layer
+   classes/adapter_model_interface
    classes/model_mixins
    classes/adapter_training
    classes/adapter_utils
@@ -110,6 +111,7 @@ Currently, we support the PyTorch versions of all models as listed on the `Model
    contributing
    contributing/adding_adapter_methods
    contributing/adding_adapters_to_a_model
+   extending
 
 Citation
 ========

docs/model_overview.md

@@ -13,6 +13,7 @@ The table below further shows which model architectures support which adaptation
 
 | Model                                   | (Bottleneck)<br> Adapters | Prefix<br> Tuning | LoRA | Compacter | Adapter<br> Fusion | Invertible<br> Adapters | Parallel<br> block | Prompt<br> Tuning | ReFT |
 | --------------------------------------- | -| - | - | - | - | - | - |- | - |
+| [Custom models](plugin_interface.html)    | ✅(°) |  | ✅ | ✅ | ✅ | ✅ |  | ✅ | ✅ |
 | [ALBERT](classes/models/albert.html)    | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 | [BART](classes/models/bart.html)        | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ✅ |
 | [BEIT](classes/models/beit.html)        | ✅ | ✅ | ✅ | ✅ | ✅ |  |  | ✅ | ✅ |
@@ -38,9 +39,11 @@ The table below further shows which model architectures support which adaptation
 | [XLM-RoBERTa](classes/models/xlmroberta.html) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 | [X-MOD](classes/models/xmod.html) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 
+(°) `original_ln_after=False` is unsupported for bottleneck configs.  
 (*) If the used encoder and decoder model class are supported.
 
-**Missing a model architecture you'd like to use?**
-adapters can be easily extended to new model architectures as described in [Adding Adapters to a Model](https://docs.adapterhub.ml/contributing/adding_adapters_to_a_model.html).
+**Missing a model architecture you'd like to use?**  
+The new model plugin interface makes it easy to support new model architectures. [Learn more](plugin_interface.md).
+Also, _Adapters_ can be extended to new model architectures as described in [Adding Adapters to a Model](https://docs.adapterhub.ml/contributing/adding_adapters_to_a_model.html).
 Feel free to [open an issue](https://github.com/Adapter-Hub/adapters/issues) requesting support for a new architecture.
 _We very much welcome pull requests adding new model implementations!_

docs/plugin_interface.md

@@ -0,0 +1,94 @@
+# Custom Models
+
+The _Adapters_ library provides a simple mechanism for integrating adapter methods into any available _Transformers_ model - including custom architectures.
+This can be accomplished by defining a plugin interface instance of [`AdapterModelInterface`](adapters.AdapterModelInterface).
+The following example shows how this looks like for Gemma 2:
+
+```python
+import adapters
+from adapters import AdapterModelInterface
+from transformers import AutoModelForCausalLM
+
+plugin_interface = AdapterModelInterface(
+    adapter_methods=["lora", "reft"],
+    model_embeddings="embed_tokens",
+    model_layers="layers",
+    layer_self_attn="self_attn",
+    layer_cross_attn=None,
+    attn_k_proj="k_proj",
+    attn_q_proj="q_proj",
+    attn_v_proj="v_proj",
+    attn_o_proj="o_proj",
+    layer_intermediate_proj="mlp.up_proj",
+    layer_output_proj="mlp.down_proj",
+)
+
+model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it")
+adapters.init(model, interface=plugin_interface)
+
+model.add_adapter("my_adapter", config="lora")
+
+print(model_2.adapter_summary())
+```
+
+## Walkthrough
+
+Let's go through what happens in the example above step by step:
+
+**1. Define adapter methods to plug into a model:**  
+The `adapter_methods` argument is the central parameter to configure which adapters will be supported in the model.
+Here, we enable all LoRA and ReFT based adapters.
+See [`AdapterMethod`](adapters.AdapterMethod) for valid options to specify here.
+Check out [Adapter Methods](methods.md) for detailed explanation of the methods.
+
+**2. Define layer and module names:**  
+While all Transformers layers share similar basic components, their implementation can differ in terms of subtleties such as module names.
+Therefore, the [`AdapterModelInterface`](adapters.AdapterModelInterface) needs to translate the model-specific module structure into a common set of access points for adapter implementations to hook in.
+The remaining attributes in the definition above serve this purpose.
+Their attribute names follow a common syntax that specify their location and purpose:
+- The initial part before the first "_" defines the base module relative to which the name should be specified.
+- The remaining part after the first "_" defines the functional component.
+
+E.g., `model_embeddings` identifies the embeddings layer (functional component) relative to the base model (location).
+`layer_output_proj` identifies the FFN output projection relative to one Transformer layer.
+Each attribute value may specify a direct submodule of the reference module (`"embed_token"`) or a multi-level path starting at the reference module (`"mlp.down_proj"`).
+
+**3. (optional) Extended interface attributes:**  
+There are a couple of attributes in the [`AdapterModelInterface`](adapters.AdapterModelInterface) that are only required for some adapter methods.
+We don't need those in the above example for LoRA and ReFT, but when supporting bottleneck adapters as well, the full interface would look as follows:
+```python
+adapter_interface = AdapterModelInterface(
+    adapter_types=["bottleneck", "lora", "reft"],
+    model_embeddings="embed_tokens",
+    model_layers="layers",
+    layer_self_attn="self_attn",
+    layer_cross_attn=None,
+    attn_k_proj="k_proj",
+    attn_q_proj="q_proj",
+    attn_v_proj="v_proj",
+    attn_o_proj="o_proj",
+    layer_intermediate_proj="mlp.up_proj",
+    layer_output_proj="mlp.down_proj",
+    layer_pre_self_attn="input_layernorm",
+    layer_pre_cross_attn=None,
+    layer_pre_ffn="pre_feedforward_layernorm",
+    layer_ln_1="post_attention_layernorm",
+    layer_ln_2="post_feedforward_layernorm",
+)
+```
+
+**4. Initialize adapter methods in the model:**
+Finally, we just need to apply the defined adapter integration in the target model.
+This can be achieved using the usual `adapters.init()` method:
+```python
+adapters.init(model, interface=adapter_interface)
+```
+Now, you can use (almost) all functionality of the _Adapters_ library on the adapted model instance!
+
+## Limitations
+
+The following features of the _Adapters_ library are not supported via the plugin interface approach:
+- Prefix Tuning adapters
+- Parallel composition blocks
+- XAdapterModel classes
+- Setting `original_ln_after=False` in bottleneck adapter configurations (this affects `AdapterPlusConfig`)

src/adapters/interface.py

@@ -15,7 +15,7 @@ class AdapterMethod:
 
     Attributes:
         bottleneck: Adapter methods using bottleneck layers.
-        prefix_tuning: Adapters methods based on Prefix Tuning.
+        prefix_tuning: Adapters methods based on Prefix Tuning. Note that this is currently unsupported via AdapterModelInterface.
         lora: Adapter methods based on low-rank adaptation.
         prompt_tuning: Adapter methods based on Prompt Tuning.
         reft: Adapters methods based on Representation Fine-Tuning.

tests/test_methods/test_on_custom_interface.py

@@ -1,9 +1,5 @@
-import tempfile
-
-import torch
-
 import adapters
-from adapters import AdapterModelInterface, AdapterSetup, DoubleSeqBnConfig, LoRAConfig, ParBnConfig, load_model
+from adapters import AdapterModelInterface, DoubleSeqBnConfig, LoRAConfig, ParBnConfig
 from transformers import Gemma2ForCausalLM, Gemma2ForSequenceClassification
 from transformers.models.gemma2.configuration_gemma2 import Gemma2Config
 from transformers.testing_utils import torch_device