# YAME — agent reference # Canonical: https://zhou-lab.github.io/YAME/llms.txt # Human guide: https://zhou-lab.github.io/YAME/ # Source: https://github.com/zhou-lab/YAME # For exact flags, always run: yame -h ## What it is YAME (Yet Another Methylation Encoder) is a C toolkit for compact CX/BGZF storage and bitwise processing of DNA methylation data. A file stores rows, not coordinates: row i is defined by a reference row space. Arrays and whole-genome CpG tracks therefore use the same pack/subset/summary operations. Used by kycg, sesame-cli and methscope-cli; `libyame.a` and the verified asset store are shared across those tools. ## Install/build conda install -c zhou-lab -c conda-forge yame Use zhou-lab rather than the older bioconda build for `fetch` and reference resolution. Source build: `make` (requires libcurl); `make CURL=0` intentionally builds without network fetching. For recent behavior use `~/repo/YAME/yame` or `yamedev`, not a copied `yame` on PATH. ## Data model A record is an 8-byte signature, a 1-byte format code, the row count n, then data packed per that code, inside BGZF frames. Records concatenate, so a cohort is one file; the `.idx` beside it maps sample name -> BGZF offset, which is what makes `subset`/`unpack` by name possible without decompressing the rest. Position is never stored. Two files are comparable only if they share a row space, so an intersection is a bitwise AND rather than a genomic join, and a row count is an identity — which is why `-R` and `-m` accept a name instead of a path, and why some are inferred from the input's row count. ## Row spaces (row count = identity) hg38 29,401,795 EPIC 866,553 MM285 287,692 mm10 21,867,837 HM450 486,427 MSA 284,309 mm39 21,889,506 EPICv2 937,690 Mammal40 38,607 HM27 27,722 ## Asset store and fetch Nothing ships with the binary: no coordinates, no feature sets, no examples. Root precedence: `-d` / tool-specific environment / `$YAME_DATA_HOME` / `${XDG_DATA_HOME:-$HOME/.local/share}/yame`. Paths mirror browser names, e.g. `/hg38/data`, `/hg38/KYCG`, `/EPIC`. yame fetch # tree browser (terminal only) yame fetch -l # registry TSV; same when piped yame fetch hg38/data # browser path; asks on a terminal yame fetch -g chromatin hg38/KYCG # facet filter (name/source/ # collection/title/database) yame fetch -n hg38/models # dry-run plan, exits 0 yame fetch -l -g methscope hg38 # the same set as TSV yame fetch -y hg38/data # scripted/non-interactive yame fetch -c hg38/data/human_hg38_test.cg # one named file, no -y needed yame fetch -u -s -o # single-file form A name is a scope: `hg38` takes the unit and all beneath it, `hg38/data` one directory, `hg38/data/test.cg` one file. Several names may be given, separated by spaces or commas. Options are accepted before or after the names, and the documented form leads with the options. Others: `-d ` store root, `-t ` upstream tag override (needs `-k` to accept an unpinned tag), `-q` no progress. Off a terminal a fetch of a DIRECTORY will not proceed without `-y` — it prints the plan and exits with "Refusing to fetch ... Re-run with -y", so a scripted or agent fetch of a folder needs `-y` (or `-n` to plan only). A name that picks out one file is its own confirmation and runs as it stands. `-l` prints ten columns: target, source, tag, store_path, file, bytes, sha256, dir_state, local, description. `-l` takes the same names a fetch does, one or several, file or directory, and lists exactly what fetching them would take, so it is the offline dry run for any fetch command. It reaches no network: the registry is compiled in. Downloads verify the pinned `SHA256SUMS` anchor and each file digest; `.part` files are renamed only after verification. `-c` writes to cwd without a manifest. `-f` re-downloads what is present and overwrites a store populated from an unrelated tag; known earlier registry tags upgrade automatically. `.idx`/`.tbi` companions are fetched with their data. Verify by hand with `shasum -a 256 -c SHA256SUMS` in any store directory. A failed download names the cause: `download failed: HTTP 404 Not Found: `, or curl's own text (`curl: Couldn't resolve host name`) for a failure below HTTP. HTTP 429 and 503 are retried up to three times, waiting what the server's `Retry-After` header asks for or 30 s then 60 s, with each wait announced on stderr even under `-q`; a `Retry-After` above 120 s is reported instead of slept. Nothing else is retried. Under `YAME_ASSETS_MIRROR` the URL in the message is the mirror's, so a mirror that is down is named rather than the upstream. Models are in the shared registry (`yame fetch hg38/models` or `mm10/models`). `hg38_wg.updecx` is a stable filename, not a stable model version — its bytes changed again at model tag v10 — so cite `zhou-lab/methscope@v10` plus SHA-256, or the YAME version that pins them, v1.45. A model can also be WITHDRAWN: v10 dropped the two routing-tree classifiers (`hg38_celltype.clfx`, `mm10_celltype_brain.clfx`) in favour of the bank classifiers published at v9. A store filled at an earlier tag keeps those files and nothing deletes them; they simply stop being listed, and `yame_store_state` reports them as NOT_LISTED rather than current. ## Formats fmt data pack -f on disk NA 0 binary yes/no b or 0 1 bit/row none 1 one byte per row c or 1 3 B per RLE run none 2 categorical states s or 2 key table + runs a label 3 M/U counts m or 3 1/2/8 B/site M=U=0 4 beta / fraction n or 4 4 B/value negative float 6 set + universe (2-bit) d or 6 2 bits/row 00 7 CpG coordinates (.cr) r or 7 delta varint n/a Format 5 is obsolete: still decodable by `unpack`, no longer packable. fmt3 on disk is tagged by the low two bits: 01 = 1 byte (M,U <= 6), 10 = 2 bytes (M,U <= 126), 11 = 8 bytes (M,U <= 2^31-1), 00 = a 14-bit run of M=U=0. fmt6 codes: 00 outside universe, 10 in universe/0, 11 in universe/1 — the universe bit is what separates "not measured" from "measured and zero", so a query and its background travel in one file. fmt6 has multiple producer-defined meanings (set/universe, binarized methylome, differential); identify the producer before interpreting it. fmt7 is a coordinate stream, not a row-packed state file. ## Commands Text in and out pack -f [-u 1-8] [-v] [out.cx] `-f` is required; there is no default. Input is one row per CpG, already in the reference's order — nothing is sorted or matched. Tab-separated columns; extra columns are extra samples. Input may be gzipped or `-` (stdin); omitting out.cx writes stdout (no index). `-u` is the fmt3 inflated unit (default 8; 1 leaves 4 bits apiece). Per-format input: -fm "MU"; -fd "setuniverse" (set must be exactly "1"); -fn a float or NA; -fs the whole line as a label; -fb 0 or 1; -fc one char; -fr BED (only chrom and start are read). unpack [options] [sample ...] -a all records, one COLUMN per record (a matrix, not stacked rows; -C names the columns); -l ; -H first N; -T last N (index). -R prepend coordinates (never inferred here); -r 0 chrm/beg0/end1 (cg-style, default), 1 chrm/beg0/end0 (allc), else chrm_beg1. -C header line. -u unit override. -f : REQUIRED for fmt3, which has no default print mode -- unpack refuses rather than guess: 0 packed (M<<32)|U (raw storage, not a beta), <0 MU, >0 beta with NA below coverage N; fmt6 — 0 prints 0/1 with NA as '2', <0 valueuniverse, >0 the raw 2-bit code. -c chunked printing, -s chunk size (not for fmt7). chunkchar [-s size] [-v] split text before packing Inspect and index info [-1] ... File Sample NSample Nrow Format UnitBytes Keys, one row per record; -1 reports only the first record of each file — the cheap way to see what a directory holds index [-s list] [-1 name] [-c] writes .idx; -c to console split [-s list] [-v] One file per record. Same raw copy as subset when the input is indexed and its records are block-aligned; -v says which path ran. hprint [-r reg | -R ref] [-c] [-g] [-s names] [-l W] [-t N] [-w N] -s names the records, one per line, in order, taking the same file `index -s` takes. Names live in the .idx sidecar, which a PIPE does not carry, so this is how a streamed view gets labels; a count that disagrees with the records is an error. Without -s the plain dump has NO label column, so its byte offsets are unchanged. -c never colour. Colour is on only when stdout is a TERMINAL, and NO_COLOR or TERM=dumb turns it off as well, so redirecting or piping gives plain text without asking. -r chr16 or chr16:10000000-10100000, reference inferred from the row count; -R gives the whole-genome window view; neither = full-dataset dump, one glyph per row with no windowing (fmt0/3/4/6, fmt6 keeping its 1/0/2 alphabet). -c disables color, -g prints deciles, -w caps columns (wider regions are window-averaged, not truncated). fmt6 blocks/shades, fmt3/4 H/M/L, fmt0 1/0. Subset subset [-o out.cx] [-l list] [-s] [-H N] [-T N] [-z 0-9] [-v] [name ...] By sample name (needs the index) or, with -s, one fmt0 bitset per requested fmt2 state. -o also writes an output index. Records come out in **list order, not store order** — as does `unpack`. A subset copies whole records unchanged, so when every requested record starts on a BGZF block boundary its compressed bytes are moved verbatim — no decompression, no re-compression, roughly 20x faster. Anything unaligned falls back to decode/encode. -z forces a re-encode at that zlib level (and so turns the copy off); -v says which path ran. rowsub [mode] > out.cx (always stdout) -l 1-based row indices; -L "chrm_beg1" lines; -R coordinates, optional with -L/-1 (inferred from the row count); -1 emit the subsetted coordinates as the first record; -m keep rows with bit 1 (mask must be fmt0 or fmt1); -B beg0[_end1] half-open block; -I blockIdx0[_blockSize] (default 1000000). Precedence: -l/-L > -m > -B/-I. Order is preserved and no sorting is required; fmt2 keys survive the slice. chunk [-s size] [-v] [outdir] fixed row blocks; fatal on fmt2/fmt7 Compare and summarize summary [-m mask|-b] [options] [query2.cx ...] Queries may be fmt 0/1/2/3/4/6/7 and multi-sample. -m accepts a name resolved in the query's own row space (`-m CGI`, `-m ChromHMM`); -b picks masks in the browser and fetches what is missing. Every query sample is summarized against every mask sample. A fmt2 mask emits one row per state. With no mask, Mask is 'global'. Columns (-V set, default; also -V 2bit): QFile Query MFile Mask N_univ N_query N_mask N_overlap Log2OddsRatio Beta Depth. -V meth (all records must be fmt6): QFile Query MFile Mask N_covered N_meth N_covered_in_mask N_meth_in_mask Log2OddsRatio Beta Beta_bg. -M preload masks (for a mask file on slow IO; it does not change speed -- see below), -H no header, -F full paths, -T always name states, -s rename query samples, -q names a stdin query, -6 deprecated alias for -V 2bit. A query with N masks reads the query ONCE, not N times: one bitmap of its covered rows is built and every mask is measured against it 64 rows at a time. Against the 1359-set TFBS knowledgebase on a 29.4M-row methylome that is 193 s -> 6.1 s for a fmt3 query and 110 s -> 5.4 s for a fmt6 one, byte-identical output, peak memory unchanged at 45 MB. fmt3 and fmt6 queries in the default view, with binary or fmt6 masks; anything else takes the old per-mask path and gives the same numbers. The two query formats measure DIFFERENT things -- a fmt6 query takes its universe from the query and from the mask, restricts every count to it, and derives beta as N_overlap/N_mask rather than as a mean of betas -- so they are separate code, not one path with a branch. Setting YAME_SUMMARY_PATH=1 reports on stderr how many masks took the one-pass kernel and how many fell back; both produce the same numbers, so that line is the only way to tell which ran. Name resolution for -m (and -R elsewhere) takes exactly three forms: a real filesystem path, the row space's own name (`hg38` -> its .cr), or a BARE filename searched in the directories that row space owns — genome first, then its knowledgebase. It picks the newest match, so `-m Win100k` finds Win100k.20220228.cm and the datestamp is never needed. It does NOT take a browser path: `-m hg38/KYCG/Win100k` fails, because the slashes make it look for a file of that literal name. Note the asymmetry — `yame fetch` wants exactly that browser spelling, so the same string fetches a set and fails to resolve it. pairwise [-S] [-1 name] [-2 name] [-t beta] [-o out] [-c cov] [-d delta] [-H 1|2|3] [B.cx] Two samples on the same row space, scored only where BOTH cover the site. fmt3 or fmt4 on either side (fmt4 covers a site when it holds a value; fmt3 when M+U >= -c, default 1). Default output: one fmt6 record. Universe = both covered; set = direction (-H 1 hyper in sample 1, 2 hypo, 3 either) and |beta1-beta2| > -d (default 0). The delta magnitude is not stored. -S instead prints one TSV line per pairing, with a header: sample1 sample2 n mae rmse acc pearson -- acc is the share of sites where both betas fall on the same side of -t (default 0.5). -1/-2 name the record to take from each file through its index; without them the first record is used, and without B.cx the second record of A.cx is sample 2. With -S and B.cx unnamed, EVERY record of B.cx is scored against sample 1 (sample 1 inflated once, B streaming) -- the one-against-many case. Two multi-record files with neither side named is an error, not a guess about whether the pairing is positional or cartesian. To score imputation, blank the sites the model was shown first: yame mask truth.cg query.cg | yame pairwise -S -1 - pred.cg rowop -o [-c mincov] [-t N] [-d 0-9] [out] binasum fmt0/1/3 -> fmt3 per-row sample counts (-p/-q beta cutoffs, default 0.4/0.6; betas between them are ignored) — this is how a pseudobulk is made musum fmt3 -> one fmt3 record, counts summed stat fmt3 -> text, NINE columns in this order: count mean_beta sd_beta delta_beta min_n delta_mean q95_0 q05_1 delta_q q95_0 = 95th pct of the beta<0.5 group, q05_1 = 5th pct of the beta>0.5 group, delta_q = q05_1 - q95_0. The quantiles come off a 16-bin histogram, so they are bin EDGES: a cutoff between edges acts as the nearer one, and `q95_0 <= 0.15` behaves as `<= 0.125`. They tolerate an outlier per side only once a group holds more than 20 samples; below that they collapse like delta_beta. binstring fmt3 -> text row-wise binary strings (-b threshold 0.5, -m max ambiguous fraction, -M min majority fold) cometh fmt3 -> text packed UU/UM/MU/MM per neighbor offset (-w window 5, -v unpacked) -t N split the records over N threads for binasum, musum and stat; the other ops stay serial and say so. Output is byte-identical at every N (the accumulators are integer, so the merge is exact). An indexed input seeks per thread, a stream is read once and dispatched. Memory is N x 8 bytes/row for binasum/musum and N x 84 bytes/row for stat — at hg38 scale 235 MB and 2.5 GB per thread, so `-t 8 -o stat` genome-wide wants ~20 GB. -d 0-9 decimals for stat's fractions (default 6). It was 3 before v1.37, and 3 puts many values on a rounding boundary, since a beta is a ratio of small integers. binarize [-t 0.5] [-m Mmin] [-c cov] [-o out] fmt3 -> fmt6. set = beta > -t, or M >= -m when -m > 0 (overrides -t); universe = coverage >= -c. Perturb and mask (benchmarking) mask [-o out] [-c] [-v] The mask is positional, not -m, and row counts must match. Data may be fmt 0/1/3/6; the mask must reduce to fmt0 (fmt1/fmt3 are converted, fmt3 by coverage M+U>0). -c contextualizes fmt0 data to fmt6 (implicit for fmt6 input), -v inverts the mask. Deterministic and file-driven: it decides which sites are valid. dsample [-N 100] [-s seed] [-r reps] [-p prefix] [-b] [-o out] fmt3 and fmt6 only. fmt3 masks by setting M=U=0, fmt6 by clearing the universe bit. Stochastic: fix -s or it is not reproducible. perturb [-p 0.05] [-s seed] [-o out] fmt0 and fmt6 only. fmt6 flips only in-universe set bits; NA is left alone. Fix -s. Reference data fetch see the store section above ## Recipes ## look at a shipped example (-y: nothing fetches unattended without it) yame fetch -y -c hg38/data/human_hg38_test.cg yame fetch -y hg38/cpg_nocontig.cr # coordinates, into the store yame unpack -R hg38 human_hg38_test.cg | awk '$4!=2' | head ## fmt6 round trip (verified byte-identical) yame unpack -f -1 human_hg38_test.cg > rt.txt yame pack -f6 rt.txt rt.cg ## per-CpG M and U for a region, with coordinates yame rowsub -R hg38 -L coords.txt -1 in.cg | yame unpack -a -f -1 - ## enrichment against a knowledgebase set yame fetch -y -g CGI hg38/KYCG yame summary -m CGI human_hg38_test.cg ## pseudobulk from single cells yame subset -l cluster_ids.txt -o cluster.cg cells.cg yame rowop -o binasum cluster.cg pseudobulk.cg ## Traps - `unpack` on fmt3 requires `-f`, and refuses without it: M/U counts have no one obvious print mode, and the old default (`-f 0`, the packed uint64 `(M<<32)|U`) looked like garbage to everyone who did not ask for it. Use `-f -1` for M and U columns, or `-f ` for beta with NA below coverage N. Every other format keeps its default. On fmt6, the default prints NA as '2'. - `pack -f1` keeps only the first character; `pack -f0` treats any line not starting with `0` as one, so `NA` is not missing; `pack -f6` needs exactly `1` in the first column to count as in-set. - Format-3 counts are compact and lossy beyond 2^31-1: `fitMU` right-shifts M and U together, so the ratio survives and the absolute depth does not. - `pack -fr` reads chromosome and start only, and starts a new block on backwards coordinates; it does not sort. - `rowsub -B beg_end` is half-open `[beg,end)`; `-l` is 1-based, `-B`/`-I` are 0-based. - `chunk` (and any block slice) is fatal on fmt2 and fmt7 — use `rowsub`. - The index is `.idx`; some older help text calls it `.cxi`. - `mask` takes its mask as a positional argument, not `-m`. - CX files do not contain coordinates. Supply the correct `-R`/`-m` reference where required; YAME can infer references only from unique known row counts, and `unpack -R` is never inferred. - A `.cg` with an index needs its companion `.idx`; use registry fetches rather than manually copying only the data file. - Records written by a command that streams several of them through one writer (`subset`, `rowsub`, `binarize`, `dsample`, `perturb`, `mask`) now start on a block boundary, which is what lets a later `subset`/`split` copy them without decoding. Files written before that change are still read normally, just not copied; one pass through `yame subset` realigns a store. - `rowop -o stat` emits **nine** columns since v1.37; it was six before q95_0, q05_1 and delta_q were added. Anything indexing it positionally — especially after a `paste` that puts other columns behind it — must be re-checked, and is better off reading the header than counting. - `subset -l` and `unpack -l` emit records in **list order**, not store order. That is the useful behaviour, but `yame index -s names.txt` pairs names to records **positionally**, so a name list in store order applied to output in list order silently mispairs every sample with someone else's data. - `yame fetch` with no target opens the browser on a terminal but dumps the registry TSV when piped or redirected, so a script never blocks. Off a terminal a fetch still refuses to download without `-y`. `NO_COLOR` disables ANSI output. ## Development cd ~/repo/YAME && make ./yame -h ./tools/make_registry.sh --tool=yame -o src/registry.h ./tools/make_assetinfo.sh -o src/assetinfo.h `src/registry.h` and `src/assetinfo.h` are generated: edit catalog/source inputs, then regenerate. The org-mode journal is the canonical release and provenance record. Do not hand-edit generated headers.