tomaarsen/mpnet-base-gooaq-mnbrl-hn-rerun-no-intra-dd

MPNet base trained on GooAQ triplets using MultipleNegativesBidirectionalRankingLoss with hard negatives

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the gooaq-hard-negatives dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: microsoft/mpnet-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • gooaq-hard-negatives
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/mpnet-base-gooaq-mnbrl-hn-rerun-no-intra-dd")
# Run inference
queries = [
    "negative electrode is anode or cathode?",
]
documents = [
    'The anode is the electrode where electricity moves into. The cathode is the electrode where electricity is given out or flows out of. The anode is usually the positive side. A cathode is a negative side.',
    'In a galvanic (voltaic) cell, the anode is considered negative and the cathode is considered positive. This seems reasonable as the anode is the source of electrons and cathode is where the electrons flow. However, in an electrolytic cell, the anode is taken to be positive while the cathode is now negative.',
    'By definition: The anode is the electrode where oxidation (loss of electrons) takes place (metal-A electrode); in a galvanic cell, it is the negative electrode, because when oxidation occurs, electrons are left behind on the electrode.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.9344, 0.9138, 0.9238]])

Evaluation

Metrics

Information Retrieval

• Dataset: gooaq-dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.589
cosine_accuracy@3	0.723
cosine_accuracy@5	0.776
cosine_accuracy@10	0.788
cosine_precision@1	0.589
cosine_precision@3	0.241
cosine_precision@5	0.1552
cosine_precision@10	0.0788
cosine_recall@1	0.589
cosine_recall@3	0.723
cosine_recall@5	0.776
cosine_recall@10	0.788
cosine_ndcg@10	0.6951
cosine_mrr@10	0.6644
cosine_map@100	0.6649

Training Details

Training Dataset

gooaq-hard-negatives

• Dataset: gooaq-hard-negatives at 87594a1
• Size: 9,000 training samples
• Columns: question, answer, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 8 tokens mean: 11.69 tokens max: 22 tokens	min: 16 tokens mean: 59.92 tokens max: 155 tokens	min: 13 tokens mean: 59.32 tokens max: 145 tokens	min: 13 tokens mean: 59.07 tokens max: 139 tokens	min: 12 tokens mean: 59.64 tokens max: 187 tokens	min: 12 tokens mean: 59.08 tokens max: 139 tokens	min: 13 tokens mean: 59.26 tokens max: 133 tokens

• Samples:

  question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  what is the difference between clay and gel?	It has a stronger hold like gel, without the high-shine finish. Clay is great for shorter spikier styles and has great "tack" in the hair. Gel leaves your hair styled, and it will more or less stay in that style until you wash it out. Gel has a glossy, high-shine look associated with it.	You're going to enjoy your experience more if you buy a quality, brand name polymer clay. So whether you've always called it Sculpty, Sculpey, Fimo, polyclay, or just plain old polymer clay, it's all the same type of material.	shower gel and bath gel are basically the same thing. but usually if something says bath on it, it doesn't work up as big of a lather then something that says shower. ... They're all just forms of soap to make it easier and cleaner to take a shower or bath. The only difference is the price tags.	"Acrylic is liquid and powder, gel is gel. ... If they paint your nail with thick goo out of a pot and then stick your hands under UV, it's gel. If they mix liquid and powder and mush it on, it's acrylics. And if they paint your nail and then sprinkle powder on, it's dip.	"Acrylic is liquid and powder, gel is gel. ... If they paint your nail with thick goo out of a pot and then stick your hands under UV, it's gel. If they mix liquid and powder and mush it on, it's acrylics. And if they paint your nail and then sprinkle powder on, it's dip.	as indicated by the two previous answers, bentonite is generally used to describe clay in the context of sedimentary geology. ... Smectite is defined in clay mineralogy as a 2:1 clay - consisting of an octahedral sheet sandwiched between two tetrahedral sheets.
  is bhop script legal csgo?	Scrips are legal in mm but not in most leagues, bhop scripts are not normal scripts, I wouldn't risk it as some might be vac bannable , macros are not vac bannable, however, it is possible to get overwatch banned from it.	Available for Steam for Mac, Windows, and Linux, the free version of CS:GO requires 16 GB free space. Valve has just announced a free version of their vastly popular first-person shooter, Counter-Strike: Global Offensive or commonly called CS: GO.	MyShopify is the extension given by Shopify whenever a merchant opens a webshop on Shopify, thus it is legit.	That site is not selling legitimate licenses. ... They are genuine Microsoft product keys, they are actually retail licenses, but it is intended for a particular product channel either the Microsoft Software Developer Network (MSDN) or TechNet for IT Professionals who pay a subscription fee.	BDoc (Business Document) messages are used in SAP CRM systems as containers for the data that constitute a business process (application message, transaction). ... Instead, the business data is packed into containers during BAPI calls.	Having a valid CSCS card is not currently a legal requirement to work as a scaffolder in the UK, but it certainly is beneficial. As time goes on, many more of the major contractors are making it a requirement for workers to have a CSCS qualification, if they wish to work on one of their sites.
  what is the difference between a simple stain and a differential stain?	A differential stain is a specific type of staining that allows for microbe identification, and distinguishing between cells in a mixed sample. This is very different from simple staining techniques that simply add color and contrast to a slide. Simple staining involves adding a basic, cationic dye to the organism.	The advantage of using stains to look at cells is that stains reveal these details and more. The arrangement of cells within a tissue reveals the health of that tissue. Multiple stains can used simultaneously to mark different cells by different colors.	Fungi, including yeast, may also be detected with a Gram stain. A Gram stain is typically ordered along with a culture when a bacterial or sometimes fungal infection is suspected. It is also usually performed when the result of a culture is positive, on a sample of the bacteria grown in the culture.	The biggest difference between concrete stain and paint is how they cover concrete. When you use indoor or outdoor concrete paint, it stays right on top of the concrete, where it provides a layer of protection. ... On the other hand, stain actually soaks into the concrete.	large amounts of cells in a smear can cause staining artifacts because stain is not washed away by destaining agents or water. ... heat fixation causes cells to adhere to the slide during staining. However, structures such as capsules can undergo shrinkage during heat fixations and can be lost in staining procedures.	Thus, commonly used basic dyes such as basic fuchsin, crystal violet, malachite green, methylene blue, and safranin typically serve as positive stains. On the other hand, the negatively charged chromophores in acidic dyes are repelled by negatively charged cell walls, making them negative stains.

• Loss: MultipleNegativesBidirectionalRankingLoss with these parameters:

  {
      "temperature": 0.01,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

gooaq-hard-negatives

• Dataset: gooaq-hard-negatives at 87594a1
• Size: 1,000 evaluation samples
• Columns: question, answer, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 8 tokens mean: 11.74 tokens max: 24 tokens	min: 15 tokens mean: 58.82 tokens max: 139 tokens	min: 12 tokens mean: 58.98 tokens max: 157 tokens	min: 14 tokens mean: 58.61 tokens max: 166 tokens	min: 14 tokens mean: 58.89 tokens max: 173 tokens	min: 14 tokens mean: 59.26 tokens max: 124 tokens	min: 15 tokens mean: 59.35 tokens max: 135 tokens

• Samples:

  question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  does your blood pressure go up after you eat?	Summary. A person's blood pressure typically drops slightly following a meal. However, foods high in sodium can cause a temporary increase in blood pressure, while foods high in saturated fat can cause longer-term issues.	Low blood pressure after eating (postprandial hypotension). Blood flows to your digestive tract after you eat. Ordinarily, your body increases your heart rate and constricts certain blood vessels to help maintain normal blood pressure. But in some people these mechanisms fail, leading to dizziness, faintness and falls.	Eating does cause changes in blood flow, which can result in an increased heart rate. Eating can also cause an increase in blood pressure. If you overeat, you force your heart to work harder than normal. You need more blood going to your digestive system, which causes your heart rate to go up.	Eating does cause changes in blood flow, which can result in an increased heart rate. Eating can also cause an increase in blood pressure. If you overeat, you force your heart to work harder than normal. You need more blood going to your digestive system, which causes your heart rate to go up.	When a person eats, their body redirects blood to the digestive tract to aid with digestion. This causes a temporary decrease in blood pressure elsewhere in the body. To compensate, blood vessels outside of the digestive tract constrict, causing the heart to beat faster and more forcefully.	Take your blood pressure before exercise, or you might get an elevated reading. Meals. In the morning, take your blood pressure before eating, as digesting food can lower your blood pressure. If you must eat first, wait 30 minutes after eating before taking a measurement.
  how to apply for us visa pakistan?	['Pay the visa application fee.', 'Schedule your appointment on this web page. You will need the following information in order to schedule your appointment: ... ', 'Visit the U.S. Embassy or Consulate on the date and time of your visa interview.']	['Valid passport with a minimum of six (6) months prior to expiration date.', 'Two (2) completed current visa application forms.', 'Two (2) photographs.', 'Fee: U.S. passport $150.00 for single entry and $175.00 for multiple entries. Contact the embassy for other nationalities.']	Step Two: Go to https://ais.usvisa-info.com/en-gb to pay the MRV application fee and schedule an interview, or obtain information on applying for the visa by courier. Click on “Apply” if you are a first time user or “Continue” if you already have an account.	['Select your course and university (a SEVP-approved institution) and apply for it.', 'Get accepted into a school.', 'Obtain your Form I-20 [or Form DS-2019 (J-1)] from the university.', 'Pay the SEVIS fee - Pay the SEVIS I-901 fee at least three business days before your visa interview; print a receipt.']	['Make an appointment to visit the Federal Benefits Unit (FBU) office. Contact the office of the FBU at the closest US Embassy or Consulate in your country of residence. ... ', 'Make an appointment to visit to the Embassy/Consulate without an FBU office.']	['Appointment Letter – The interview appointment letter you received from NVC.', 'Passport – For each applicant, an unexpired passport valid for six months beyond the intended date of entry into the United States.']
  what is the difference between independent director and non executive director?	A non-executive director may be representing a major shareholder but an independent director will generally have no other links with the company other than sitting on the board. Non-executive directors' principal role is to provide independent judgement. ... recognising the division between the board and management.	A non-executive director typically does not engage in the day-to-day management of the organization but is involved in policymaking and planning exercises. A non-executive director's responsibilities include the monitoring of the executive directors and acting in the interest of the company stakeholders.	Shareholders and directors have two completely different roles in a company. The shareholders (also called members) own the company by owning its shares and the directors manage it. Unless the articles say so (and most do not) a director does not need to be a shareholder and a shareholder has no right to be a director.	Executive directors and presidents of nonprofits have very different responsibilities. An executive director at a nonprofit will be in charge of the daily running of the organization. The president is the overall leader and visionary of the organization's mission.	Independent agencies are not subject to direct control by the president or the executive branch, unlike executive agencies. The leaders of independent agencies do not serve as part of the president's Cabinet.	Responsibilities of Executive Directors vs. Presidents in a Nonprofit. ... The executive director runs the day-to-day operations of a nonprofit, while the president runs board meetings, outlines the code of laws for an organization, and may sign contracts.

• Loss: MultipleNegativesBidirectionalRankingLoss with these parameters:

  {
      "temperature": 0.01,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooaq-dev_cosine_ndcg@10
-1	-1	-	-	0.2412
0.0035	1	9.5711	-	-
0.0532	15	6.3702	-	-
0.1028	29	-	2.1272	0.5498
0.1064	30	3.0429	-	-
0.1596	45	2.0515	-	-
0.2057	58	-	1.5068	0.6630
0.2128	60	1.6725	-	-
0.2660	75	1.6143	-	-
0.3085	87	-	1.3883	0.6731
0.3191	90	1.4829	-	-
0.3723	105	1.4976	-	-
0.4113	116	-	1.3255	0.6836
0.4255	120	1.5065	-	-
0.4787	135	1.4254	-	-
0.5142	145	-	1.3057	0.6879
0.5319	150	1.4758	-	-
0.5851	165	1.3577	-	-
0.6170	174	-	1.2972	0.6903
0.6383	180	1.3983	-	-
0.6915	195	1.4823	-	-
0.7199	203	-	1.2748	0.6922
0.7447	210	1.5255	-	-
0.7979	225	1.4456	-	-
0.8227	232	-	1.2829	0.6938
0.8511	240	1.3839	-	-
0.9043	255	1.3544	-	-
0.9255	261	-	1.2715	0.6955
0.9574	270	1.3751	-	-
-1	-1	-	-	0.6951

Framework Versions

• Python: 3.11.14
• Sentence Transformers: 5.3.0.dev0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu130
• Accelerate: 1.12.0
• Datasets: 4.4.2
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}