dataspires

Overview

AfriLink gives you one-line access to A100 GPUs for training (any framework — YOLOv8, custom PyTorch, etc.), finetuning (LoRA/QLoRA for LLMs), and inference (HuggingFace endpoints). The SDK handles authentication, dataset upload, job submission, status polling, model download, and per-job billing. Install with pip install afrilink-sdk.

Quick Start

pip install afrilink-sdk

from afrilink import AfriLinkClient

client = AfriLinkClient()
client.authenticate()  # DataSpires + HPC credentials

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=your_dataframe,
    gpus=1,
    time_limit="01:00:00",
    backend="cineca",
)
result = job.run(wait=True)

if result["status"] == "completed":
    client.download_model(result["job_id"], "./my-model")

    from transformers import AutoModelForCausalLM, AutoTokenizer
    from peft import PeftModel

    base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
    model = PeftModel.from_pretrained(base, "./my-model")
    tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
    out = model.generate(**tokenizer("Hello!", return_tensors="pt"), max_new_tokens=64)
    print(tokenizer.decode(out[0], skip_special_tokens=True))

Installation

pip install afrilink-sdk

Zero required dependencies — heavy libraries are only needed at the point you use them and are pre-installed in most notebook environments.

Authentication

Both phases happen inside a single client.authenticate() call:

Pass credentials explicitly: client.authenticate(dataspires_email="...", dataspires_password="...")

After auth you get an SSH certificate (~12h), SLURM job manager, SCP transfer manager, and telemetry tracker. The SDK warns before expiry — see Session Recovery.

HPC Backends

Hardware Specs — High Performance Compute Cluster

Each GPU node on the Booster partition (where AfriLink jobs run):

Per-GPU memory guide for model loading:

Billing: $2.00 / GPU-hour, charged per completed GPU-minute (minimum 1 minute). Credits deducted automatically from your DataSpires balance.

Available Models

Use client.list_available_models() to browse, or filter with size="tiny".

Training Modes

Dataset Formats

DataFrame should have a text column with the full prompt + response (Alpaca-style or chat template).

Training (General-Purpose)

Use client.train() to run any training script on HPC. Works with any framework that runs inside a Singularity container (YOLOv8, custom PyTorch, etc.). For LoRA/QLoRA LLM fine-tuning, use client.finetune() instead.

job = client.train(
    script="train_yolo.py",        # your training script
    container="afrilink-yolo",      # pre-built container
    data="./dataset/",              # uploaded automatically
    data_config="dataset.yaml",     # config file
    gpus=1,
    time_limit="02:00:00",
)
result = job.run(wait=True)
print(job.get_logs(tail=50))

Available Containers:

Data handling: Pass a local directory, .tar.gz archive, single file, pandas DataFrame, or remote HPC path (starting with $ or /). Archives are automatically extracted on the cluster.

TrainJob has the same interface as FinetuneJob: .run(), .cancel(), .get_logs(), .status, .job_id, .estimated_cost_usd().

SDK Reference

AfriLinkClient

TrainJob / FinetuneJob (returned by client.train() and client.finetune())

run() returns a dict with job_id, slurm_job_id, status, and output_dir. Always check result["status"] before downloading.

Pricing

Pay-as-you-go at $2.00 per GPU-hour. Add credits via card payment or redeem voucher codes on your Billing Dashboard.

Session Recovery

SSH certificates expire after ~12 hours. The SDK warns at 60, 30, 15, and 5 minutes before expiry. Call recover_session() to pick up where you left off:

# Re-authenticate and download completed models
recovery = client.recover_session("./recovered-models")
print(recovery.re_authenticated)   # True if fresh cert obtained
print(recovery.jobs)                # status of each tracked job
print(recovery.files_retrieved)     # downloaded model directories

# Or just re-authenticate without downloading
client.recover_session()

What it does:

1. Re-authenticates — fresh SSH certificate without re-entering credentials
2. Checks all tracked SLURM jobs and reports their current status
3. Downloads completed models automatically if you pass a download_dir
4. Registers email notification for jobs still running

Jobs keep running on the cluster after cert expiry — you just need fresh credentials to check on them.

Built-in User Guide

Query the inline reference manual from any notebook cell — no internet required:

import afrilink

afrilink/help          # index of all topics
afrilink/quickstart    # getting started
afrilink/auth          # authentication & sessions
afrilink/finetune      # finetune parameters & modes
afrilink/specs         # models and GPU requirements
afrilink/datasets      # dataset formats and upload
afrilink/transfer      # file upload & download
afrilink/jobs          # SLURM job management
afrilink/inference     # inference routing to HuggingFace

Inference

Route prompts to any HuggingFace Inference Endpoint — public or private — without an active HPC session:

# Public model
result = client.inference(
    "Explain LoRA fine-tuning in one sentence.",
    model_id="HuggingFaceH4/zephyr-7b-beta",
)
print(result.text)

# Gated model with HF token
result = client.inference(
    "What is transfer learning?",
    model_id="meta-llama/Llama-2-7b-chat-hf",
    hf_token="hf_...",
    parameters={"max_new_tokens": 256, "temperature": 0.7},
)

# Private HuggingFace Inference Endpoint
result = client.inference(
    payload={"inputs": "Hello!"},
    endpoint_url="https://xyz.endpoints.huggingface.cloud",
    hf_token="hf_...",
)

if result.success:
    print(result.text)
else:
    print(f"Error {result.status_code}: {result.error}")

Working With Your Model

Once downloaded, adapter weights work directly with standard HuggingFace tooling.

Export to GGUF & run with Ollama

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Merge adapter into base model
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
merged = PeftModel.from_pretrained(base, "./my-model").merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")

# python convert_hf_to_gguf.py ./my-model-merged --outfile my-model.gguf
# ./llama-quantize my-model.gguf my-model-q4.gguf Q4_K_M
# ollama create my-model -f Modelfile  &&  ollama run my-model

Publish to HuggingFace Hub

from huggingface_hub import HfApi

api = HfApi(token="hf_...")
repo_id = "your-username/my-finetuned-model"
api.create_repo(repo_id, exist_ok=True)

api.upload_folder(folder_path="./my-model", repo_id=repo_id)          # adapter only
api.upload_folder(folder_path="./my-model-merged", repo_id=repo_id)   # full merged model
api.upload_file(path_or_fileobj="./my-model-q4.gguf",
                path_in_repo="my-model-q4.gguf", repo_id=repo_id)       # GGUF