ZharfAI Team
AI can search literature, propose hypotheses, write analysis code, classify images, extract measurements, and draft protocols. These capabilities make exploration faster. They do not make a result scientific by themselves. The central test remains whether a knowledgeable researcher can inspect the evidence, understand the method, and obtain a materially consistent result.
The failure mode is often not a spectacular hallucination. It is a plausible table whose data version is unknown, code that ran once in an unrecorded environment, a literature summary that erased disagreement, or a generated protocol that was edited at the bench without documenting the deviation. The result looks finished while its provenance has disappeared.
The National Academies distinguishes reproducibility—obtaining consistent computational results using the same data, code, methods, and conditions—from replicability, which tests whether a new study addressing the same question obtains consistent results. Its report on Reproducibility and Replicability in Science is a useful vocabulary check. AI-assisted science needs both. Re-running an analysis can expose a broken artifact; an independent experiment can expose a fragile scientific claim.
AI is most useful when it expands the search space: suggesting mechanisms, identifying related literature, translating an idea into candidate code, or proposing controls. Label those outputs as candidates. A generated hypothesis is not evidence. A synthetic sample is not an observation. A simulation is evidence about the model under its assumptions, not automatically about the world.
Create explicit stages:
An AI assistant may operate in every stage, but the stage must be recorded. Exploration can be flexible; confirmatory evidence requires a predeclared or otherwise justified method. If an analysis was chosen after inspecting the outcome, say so. A polished narrative should not convert an exploratory result into a preregistered one.
For synthetic or augmented observations, define exactly where they enter the pipeline and compare conclusions with and without them. The synthetic-data governance guide covers leakage, representativeness, and traceability that apply directly to research use.
Saving a prompt is not enough. The reproducibility envelope is the smallest package that lets another researcher understand and rerun the AI-assisted step. It should include:
Do not record hidden chain-of-thought or confidential credentials. Record observable inputs, outputs, decisions, and validation evidence. If a hosted model does not expose an immutable version, archive the output and timestamp, record the provider's version label, and disclose that exact replay may not be possible.
The FAIR Guiding Principles apply beyond conventional datasets: the authors explicitly discuss algorithms, tools, and workflows as research objects. Findable, Accessible, Interoperable, and Reusable does not mean “public at any cost.” Metadata can remain findable while human-subject or proprietary data are accessed through a controlled process.
A language model can compress dozens of papers into smooth prose while merging different populations, endpoints, or study designs. Prevent this with a claim table. For every consequential statement, store:
| Field | What to record |
|---|---|
| Claim | The precise proposition, not a topic summary |
| Source | DOI or stable URL and exact version |
| Location | Page, section, table, or figure |
| Study context | Population, sample, intervention, comparator, endpoint |
| Evidence type | Experiment, observation, simulation, review, opinion |
| Direction and uncertainty | Effect, interval, caveat, or null result |
| Reviewer status | Extracted, checked, disputed, or excluded |
Retrieve the source text before asking the model to interpret it. Verify quotations and numerical claims against the primary paper. Search for disconfirming evidence and retractions. If sources disagree, preserve the disagreement instead of asking the model for a forced consensus.
Useful metrics include citation precision for sampled claims, unsupported-claim rate, source-retrieval success, retraction-screen coverage, and reviewer disagreement. These are more diagnostic than rating whether the prose “sounds accurate.”
AI-generated code can silently choose defaults, leak test data, mishandle missingness, invert labels, or compute a statistic different from the one described. Review it as an instrument that transforms evidence.
Suppose a model generates a pipeline comparing two diagnostic classifiers. The code imputes missing values across the full dataset before the train-test split. It runs successfully and produces an impressive area under the curve, but information from the test set has leaked into training. A reviewer must inspect the order of operations, not merely confirm that the notebook executes.
Require unit tests for transformations and metric calculations, small hand-computed fixtures, schema assertions, leakage checks, and comparison with a trusted implementation where possible. Freeze the final code before the confirmatory run. Store intermediate checksums so a reviewer can identify the first stage where results diverge.
Containers help but do not solve everything. Hardware kernels, numerical libraries, parallel execution, stochastic algorithms, and remote services can remain nondeterministic. Define numerical tolerance and expected qualitative conclusions instead of requiring bit-for-bit equality when the method cannot provide it.
The frontier-model evaluation guide offers useful patterns for frozen test sets, contamination review, and slice reporting; the same discipline belongs in scientific pipelines.
Computational provenance begins with the raw observation, not the CSV handed to an analyst. Record acquisition device, calibration, sampling process, preprocessing, exclusions, label creation, and links between original and derived files. Use checksums and immutable identifiers for released versions.
In a wet laboratory, the intended protocol and executed protocol are different records. Capture lot numbers, instrument state, timing, environmental conditions, operator-reported anomalies, and deviations. An AI-generated instruction should not overwrite what actually happened. Corrections should be appended with author and time, leaving the original record recoverable.
Human-subject data require privacy-aware reproducibility. Provide a data dictionary, synthetic fixture, analysis code, and a documented access procedure when raw data cannot be public. The NIH Data Management and Sharing Policy connects sound data management with reproducibility while recognizing legal, ethical, and technical constraints on sharing.
The best quality check is to hand the package to someone who did not build it. Give them the paper and declared artifact, but not private coaching. Ask them to create the environment, run the analysis, reproduce designated results, and document every ambiguity.
Define success before the test:
ACM's artifact review model distinguishes availability, evaluated functionality, reusability, reproduced results, and replicated results. A current conference implementation explains these artifact badges and review criteria. The distinctions matter: a public repository is not necessarily usable, and code that runs is not necessarily evidence for the paper's claim.
Track time-to-first-success, environment build success rate, proportion of claims tied to executable artifacts, reviewer interventions, and reproducibility across hardware or model versions. Publish the failed attempt when it reveals a material limitation.
Generative services complicate exact replay. Outputs can vary with sampling, hidden system changes, retrieval indexes, safety layers, and tool availability. Seeds may not control all of these. Model names can point to moving deployments.
For each AI-dependent claim, ask three questions:
Report distributions, not a favorite run. If twenty extractions produce different values, calculate agreement and adjudicate discrepancies. If a conclusion changes when one provider updates a model, label that dependency as a limitation.
For critical measurements, prefer deterministic software after the model has proposed a structure. A model may identify candidate regions in microscopy images; a versioned segmentation algorithm can then calculate area under fixed rules, with blinded human review of a sample. This division uses AI for scale without making an opaque generation the sole measurement.
Classify each step by consequence and verifiability:
| Step type | Example | Required control |
|---|---|---|
| Low consequence, easy to verify | Reformatting citations | Automated checks and sampled review |
| High consequence, easy to verify | Generated statistical code | Tests, code review, frozen execution |
| Low consequence, hard to verify | Brainstorming mechanisms | Label exploratory; do not cite as evidence |
| High consequence, hard to verify | Interpreting ambiguous pathology | Domain expert adjudication and independent method |
The dangerous quadrant is high consequence and hard to verify. AI may assist triage, but it should not become the only path from observation to claim. Either add a credible verification method or narrow the conclusion.
No. A notebook without data versions, environment, execution order, dependencies, and instructions may not run. It can also run while implementing the wrong method.
No. Ethical, legal, contractual, or security restrictions may require controlled access. Publish useful metadata, data dictionaries, code, test fixtures, and access conditions so the boundary is explicit.
Not always. Hardware, libraries, concurrency, remote APIs, and hidden model changes can introduce variation. Record the seed, environment, and acceptable tolerance, then test repeated runs.
Authorship policies vary, but a tool cannot take responsibility for the work. Disclose material AI assistance according to venue policy, and keep accountable humans responsible for verification and claims.
Terminology differs across fields and publishers, especially “replication” and “reproduction.” State the definition being used. Exact generative-model replay may remain impossible when providers do not expose immutable versions; that limitation should be disclosed rather than hidden by a seed value.
An operational view of AI in Iranian banking: fraud detection, credit scoring, Persian customer assistants, and document automation, with governance requirements and a low-risk pilot path.
Read More
An evidence-first checklist for selecting an AI company in Iran: define the workflow, test Persian performance, examine security, measure a pilot, and negotiate an exit.
Read More
Iranian AI is more than a label. A decision guide for weighing local platforms against foreign tools: data residency, access resilience, Persian quality, and a fair evaluation checklist.
Read MoreSee the daily briefing and the operational guides. This page is an archive note, not an invitation to start a project.