MSEB by Google Research: Audio encoder implementations under a benchmark contract and evaluation across four task classes
A practical walkthrough of the mseb==0.1.0 package: its architecture, built around the types, encoder, and evaluators layers; two demo encoders using a synthetic corpus; and direct calls to metrics for classification, clustering, search, and segmentation. The guide shows how to align a custom model with the abstract MultiModalEncoder, interpret the SoundEmbedding format, and assemble TaskMetadata for later submission to the benchmark.
Purpose and layers of the benchmark
MSEB — Massive Sound Embedding Benchmark by Google Research. Install the package as mseb==0.1.0. The benchmark is divided into three layers.
typesdefines theSound,SoundEmbedding,Score, andTaskMetadatatypes.encodercontains the abstract base classMultiModalEncoder— the contract implemented by a model.evaluatorscontains evaluators for different tasks.
These layers separate the description of data, the model interface, and quality measurement. Practical rule: the encoder layer defines the requirements for an encoder, while evaluators defines the requirements for evaluation.
Encoder contract and data types
encoder contains the abstract base class MultiModalEncoder. This class is the contract implemented by a model. types defines Sound, SoundEmbedding, Score, and TaskMetadata.
SoundEmbedding stores embeddings with shape (N, D) and timestamps with shape (M, 2). The timestamps specify the start and end of an interval in seconds. If M == N, the timestamps correspond to individual frames. If M == 1, the embedding applies to the entire utterance. The embedding field can contain N strings instead of vectors. SoundEmbedding also includes context and encoding statistics, including input and embedding dimensions.
Score describes the metric, its value, the allowed range, and a description. TaskMetadata is collected at the end of the evaluation and used in a real submission.
Practical rule: the encoder must return a SoundEmbedding with valid shapes and populate Score for each task.
Two encoder examples: loudness and timbre
The guide implements two different encoders. One measures loudness over time, and the other measures timbre. They are applied to a small synthetic audio corpus. The corpus is created in the notebook, so no data download is required.
According to the guide, the two encoders show different results depending on the evaluator. This means that the choice of feature affects performance across different tasks. Practical rule: both approaches should be tested for classification, clustering, retrieval, and segmentation.
Evaluation across four task classes
Embeddings are evaluated on classification, clustering, retrieval, and segmentation tasks. The examples import ClassificationEvaluator, ClusteringEvaluator, RetrievalEvaluator, and SegmentationEvaluator. Metric functions are called directly. At the end, TaskMetadata, used in a real submission, is collected.
The package also lists evaluators for reranking and transcription.
The classification, clustering, retrieval, and segmentation evaluators depend only on NumPy and scikit-learn. The reranking and transcription evaluators use Whisper.
| Task | Evaluator | Dependencies |
|---|---|---|
| Classification | ClassificationEvaluator | NumPy, scikit-learn |
| Clustering | ClusteringEvaluator | NumPy, scikit-learn |
| Retrieval | RetrievalEvaluator | NumPy, scikit-learn |
| Segmentation | SegmentationEvaluator | NumPy, scikit-learn |
| Reranking | reranking evaluator | Whisper |
| Transcription | transcription evaluator | Whisper |
Practical rule: NumPy, scikit-learn, and a CPU are sufficient for the four task classes. Reranking and transcription require Whisper.
Dependencies and runtime requirements
The steps shown run on a CPU without downloading a dataset. The classification, clustering, retrieval, and segmentation evaluators depend only on NumPy and scikit-learn. The reranking and transcription evaluators use Whisper. The task runner uses TensorFlow and apache-beam.
These dependencies determine the runtime environment. The four task classes do not require a GPU or external data. Reranking, transcription, and running the task runner require additional components.
Practical rule: if the goal is to quickly test an encoder on classification, clustering, retrieval, and segmentation, a CPU and the specified libraries are sufficient.
Practical selection criteria
An encoder implementation must conform to MultiModalEncoder. The returned SoundEmbedding must contain embeddings with shape (N, D) and timestamps with shape (M, 2). The embedding field can contain N strings instead of vectors. Score must describe the metric, value, allowed range, and description. TaskMetadata is collected at the end and used in a real submission.
For evaluation on classification, clustering, retrieval, and segmentation, use ClassificationEvaluator, ClusteringEvaluator, RetrievalEvaluator, and SegmentationEvaluator. They depend on NumPy and scikit-learn. Reranking and transcription require Whisper.
Practical rule: choose an encoder for the task. The two encoder examples — loudness and timbre — produce different results depending on the evaluator. Test both encoders on the required set of tasks.
Frequently asked questions
Related materials
All materials
Moving to the Probability Simplex Reduces the Modality Gap in Contrastive Multimodal Models
The paper arXiv:2610.32890 proposes replacing spherical embeddings with a probabilistic simplex, adding lightweight softmax adapters and Total Variation similarity. Across five retrieval benchmarks, the cross-modal centroid ℓ2 distance decreases by 97–99% while retrieval quality is maintained or improved.
3 November 2026
How ActionRAG Reconsiders Evidence During Diffusion Across the Memory Graph of LLM Agents
A graph-memory architecture is proposed that carefully reconciles entities during indexing and reevaluates relationships at query time based on local structure. On MemoryAgentBench, the method achieved an average accuracy of 55.1, outperforming the strongest retrieval baseline by 5.4 percentage points while keeping synthesis costs low per query.
3 November 2026
Controlled Evolution of LLM Agents in Autonomous Investing: A Multi-Agent Architecture with an Immutable Safety Layer
The article describes Steward, a system of nine role-based LLM agents in which portfolio decisions are reviewed by a risk manager, and prompt changes are allowed only through reversible controls and replay of recorded data. The key takeaway is that an immutable core and audit procedures allow the agents to evolve without giving them control over risk rules and order execution.
1 November 2026