Semantic search using Microsoft's Phi-2 and Spotify Annoy

Table of Contents

Key components of semantic search

Entity recognition

Contextual understanding:

Natural language processing (NLP):

Building a Semantic Search Pipeline

1. User Input:

2. Preprocessing:

3. Large Language Model (LLM):

4. Embedding:

5. Document Embedding:

6. Semantic Matching:

7. Ranking:

8. Post-Processing:

9. Presentation of Results:

10. User Feedback and Iteration:

Microsoft Phi-2

Using W&B

pip install wandb

wandb login

Dataset

table = wandb.Table(data=df)
run.log({'data':table})

Code

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True)

map = {}
archive = {}
count = 0
for index, row in df.iterrows():
    count+=1
    data = row['abstract']
    embedding = torch.zeros(384)
    emb = tokenizer(data, return_tensors="pt", max_length=384, padding='max_length', truncation=True)
    embedding[:emb.input_ids.shape[1]] = emb.input_ids[0]
    map[row['id']] = embedding
    archive[row['id']] = [row['title'], row['abstract'], row['categories']]

from annoy import AnnoyIndex

map2 = {}
index = AnnoyIndex(384)
for i, id in enumerate(map):
    index.add_item(i, map[id])
    map2[i] = id
# building the index
n_trees = 10000/100
index.build(n_trees=1)

query = "Physics"
target_vector = torch.zeros(384)
emb = tokenizer(query, return_tensors="pt", max_length=384, padding='max_length', truncation=True)
target_vector[:emb.input_ids.shape[1]] = emb.input_ids[0]

k = 3
nearest_neighbors = index.get_nns_by_vector(target_vector, k)
for i in range(k):
  print("Result ", i+1)
  print("*"*10)
  id = nearest_neighbors[i]
  for i in archive[map2[id]]:
    print(i)
    print(" ")

Results

Result 1

**********

Scientific Machine Learning through Physics-Informed Neural Networks:

Where we are and What's next

Physics-Informed Neural Networks (PINN) are neural networks (NNs) that encode

model equations, like Partial Differential Equations (PDE), as a component of

the neural network itself. PINNs are nowadays used to solve PDEs, fractional

equations, integral-differential equations, and stochastic PDEs. This novel

methodology has arisen as a multi-task learning framework in which a NN must

fit observed data while reducing a PDE residual. This article provides a

comprehensive review of the literature on PINNs: while the primary goal of the

study was to characterize these networks and their related advantages and

disadvantages. The review also attempts to incorporate publications on a

broader range of collocation-based physics informed neural networks, which

stars form the vanilla PINN, as well as many other variants, such as

physics-constrained neural networks (PCNN), variational hp-VPINN, and

conservative PINN (CPINN). The study indicates that most research has focused

on customizing the PINN through different activation functions, gradient

optimization techniques, neural network structures, and loss function

structures. Despite the wide range of applications for which PINNs have been

used, by demonstrating their ability to be more feasible in some contexts than

classical numerical techniques like Finite Element Method (FEM), advancements

are still possible, most notably theoretical issues that remain unresolved.



cs.LG cs.AI cs.NA math.NA physics.data-an

Result 2

**********

Anomalies, equivalence and renormalization of cosmological frames

We study the question of whether two frames of a given physical theory are

equivalent or not in the presence of quantum corrections. By using field theory

arguments we claim that equivalence is broken in the presence of anomalous

symmetries in one of the frames. This is particularized to the case of the

relation between the Einstein and Jordan frames in scalar-tensor theories used

to describe early Universe dynamics. Although in this case a regularization

that cancels the anomaly exists, the renormalized theory always develop a

non-vanishing contribution to the S-matrix that is present only in the Jordan

frame, promoting the different frames to different physical theories that must

be UV completed in a different way.



hep-th gr-qc

Result 3

**********

Error correction and fast detectors implemented by ultra-fast neuronal

plasticity

We experimentally show that the neuron functions as a precise

time-integrator, where the accumulated changes in neuronal response latencies,

under complex and random stimulation patterns, are solely a function of a

global quantity, the average time-lag between stimulations. In contrast,

momentary leaps in the neuronal response latency follow trends of consecutive

stimulations, indicating ultra-fast neuronal plasticity. On a circuit level,

this ultra-fast neuronal plasticity phenomenon implements error-correction

mechanisms and fast detectors for misplaced stimulations. Additionally, at

moderate/high stimulation rates this phenomenon destabilizes/stabilizes a

periodic neuronal activity disrupted by misplaced stimulations.



q-bio.NC

References

https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
https://huggingface.co/microsoft/phi-2
https://docs.wandb.ai/