KnowYourCG 2

Install — then fetch once

# both binaries, from conda
conda install -c zhou-lab -c conda-forge kycg yame

# or build from source (YAME is a submodule)
git clone --recurse-submodules \
  https://github.com/zhou-lab/kycg
cd kycg && make && make yame-bin

Install yame too. kycg links libyame statically, so the backend's library is inside the binary — but the yame command is a separate package, and the row-space check below needs it. From source, make builds only the library; make yame-bin adds external/YAME/yame.

Dependencies are YAME's — vendored htslib, zlib, libm, pthreads and librt, plus libcurl for fetch (optional; make CURL=0 builds without it) — with YAME linked statically from a pinned submodule. kycg --version (or -v) reports the coupled YAME version, the registry tag it can verify, and whether the build has libcurl.

# guided: location, collection, sets, confirm
kycg fetch

# or name what you want outright
kycg fetch mm10
kycg fetch hg38:CGI,ChromHMM

# open with hg38 checked; f to start
kycg fetch hg38

The store — fetch, verified

Knowledgebases lived in three places with three access patterns and no verification. fetch is the tie.

# -f skips the browser: plan, then fetch
$ kycg fetch -f mm10:CGI

==> Knowledgebase sets to download
    mm10  •  KYCGKB_mm10  •  ~/.local/share/yame/mm10/KYCG

    CGI.20220904.cm                                  120 KB
    cpg_nocontig.cr                                 22.7 MB

    2 file(s), 22.8 MB

✓ CGI.20220904.cm                    120 KB
✓ cpg_nocontig.cr                    22.7 MB

✓ 2 fetched (22.8 MB).

…into a plain layout keyed on the target you fetch — the address you browse to, not the repo it came from — shared with the rest of the suite. A .cm in the store is an ordinary file that kycg test -m takes by path. No database, no lockfile, no bookkeeping.

~/.local/share/yame/
├ mm10/                   # whole genome (KYCGKB_mm10)
│ ├ cpg_nocontig.cr         # the row list
│ └ KYCG/
│   ├ ChromHMM.20220414.cm
│   └ SHA256SUMS
└ MSA/                    # array (InfiniumAnnotation)
  ├ MSA.ordering.tsv.gz     # probe ordering
  └ KYCG/
    ├ CGI.20220904.cm(+.idx)
    └ SHA256SUMS

Pick what you need — r, i, -o

The full hg38 collection is 363 MB. You rarely want all of it.

# a tree — → unfolds a target in place
$ kycg fetch
    target  kind          rows        source           cached_sets
  ▸ hg38    whole genome  29,401,795  KYCGKB_hg38      3/32
❯ ▾ mm10    whole genome  21,867,837  KYCGKB_mm10     28/28
    ├ CGI       CGI…cm       120 KB  cached
    ├ ChromHMM  ChromHMM…cm  857 KB  cached
    ├ PMD       PMD…cm      16.3 KB  cached
  row 1 of 10  •  0 selected  •  → open  ← close  space select  i close  f fetch  d store   q quit

# or name it directly — opens with those checked
$ kycg fetch hg38:CGI,ChromHMM

# no browser, no questions (scripts, Docker, Nextflow)
$ kycg fetch -f hg38:CGI,ChromHMM

i opens a pane answering what a set actually is. It is on by default and follows the cursor, so arrowing down walks you through the catalogue with each set explained as you reach it — i hides it when you want the full screen for scanning:

  TFBSrm  Transcription factor binding sites, ReMap

  A larger, uniformly reprocessed TF binding compendium — around 1,188
  factors — built from published ChIP-seq rather than one lab's panel.
  The rm suffix is ReMap, not repeat-masked.

  source      ReMap 2022, non-redundant MACS2 peaks
  citation    Hammal et al. 2022, Nucleic Acids Res, doi:10.1093/nar/gkab996
  processing  remap2022_nr_macs2 peaks, one set per factor.

Test — the core

Hypergeometric tail, six effect sizes, stratified FDR. 21.8M rows in 0.12 s.

kycg test -m ~/.local/share/yame/mm10/KYCG/ChromHMM.20220414.cm \
  query.cg > res.tsv

-m is repeatable — this is what makes a store worth having. One pass, one table, one FDR correction:

kycg test $(for f in ~/.local/share/yame/mm10/KYCG/*.cm; do
        printf -- '-m %s ' $f; done) \
  query.cg > res.tsv

Omit -m on a terminal and kycg offers the store — filtered to the sets that share your query's row space, so the list cannot be picked wrong:

$ kycg test onecell.cg

Knowledgebases for 21,867,837 rows -- space to choose, t to test
        TARGET   KIND          ROWS                    TAG        IN STORE
  ▪ MOUSE
  ▾ [*] mm10     genome  21,867,837                    mixed ▰▱▱▱▱    4/29
  └── ▾ [*] KYCG        sets                           v2    ▰▱▱▱▱    3/28
      ├──   [ ] Blacklist.20220304.cm                  v2            636 B
      ├──   [x] CGI.20220904.cm                        v2         120.0 KB
      ├──   [x] ChromHMM.20220414.cm                   v2         857.4 KB
❯     ├──   [ ] ChromHMMfullStack.20231222.cm          v2           6.1 MB
  …
  row 7 of 31  •  2 selected  •  i info  f fetch  t test  h keys  q quit

It is the same browser fetch uses, showing only the collections whose row count matches the query. A set you do not have yet is still listed: check it, f to fetch it, then t to test against it without leaving. The browser is coloured whenever it draws; for plain text use -l, -y or a pipe. A terminal that cannot address the cursor (TERM=dumb) or a redirected stdout falls back to plain TSV instead.

Read the result — top hits

One row per (query sample, knowledgebase record), sorted by significance.

$ kycg test -m ChromHMM.cm -m CGI.cm -m PMD.cm \
    onecell.cg | cut -f3,4,10,13

db_file          db         estimate  fdr
PMD…cm           commonHMD  0.275     0
ChromHMM…cm      Quies      0.199     0
ChromHMM…cm      QuiesG     0.253     0
ChromHMM…cm      Tx         0.307     0
CGI…cm           Shore      0.196     9.17e-266

Check the row space — yame info

Everything is positional: row i must mean the same CpG in the query and in the knowledgebase. Confirm with yame info before you test — yame is kycg's backend but a separate command, from its own package (conda install -c zhou-lab -c conda-forge yame, or make yame-bin in a source tree). kycg itself has no info subcommand.

$ yame info onecell.cg ChromHMM.20220414.cm
File        Sample Ncol Nrow     Format UnitBytes Keys
onecell.cg  1      NA   21867837 3      1         NA
ChromHMM…cm 1      NA   21867837 2      1         N=19|Quies,…

Mismatches fail loudly, naming both files and both counts:

Row count mismatch: query 'small.cg' record '1'
has 5 rows but knowledgebase 'chromhmm.cm'
record '1' has 21867837 rows.
These files index different reference row
lists and cannot be compared.

Annotate — label probes by set

The lookup half of the same data test aggregates over: test asks whether a group of probes is enriched somewhere, annotate asks where one probe actually falls. It reads a TSV of probe IDs and writes it back with one column added per knowledgebase.

$ kycg annotate -m EPIC:CGI,ProbeType hits.tsv
# every input column and row kept, in order, plus:
Probe_ID     …   CGI     ProbeType
cg00000029   …   opensea cg
cg00000109   …   island  cg

# -i gives a 0/1 indicator column per set instead of names
$ kycg annotate -i -m EPIC:CGI hits.tsv

Plot — cinderplot, separately

kycg emits TSV; cinderplot renders it as its own process. No linking, so Cairo never enters kycg's dependency surface.

# volcano — drop the zero-overlap sentinel first
awk -F'\t' 'NR==1||($10>-1000&&$10<1000)' res.tsv > v.tsv
cinderplot 'v.tsv
  + aes(estimate, neglog10_fdr)
  + geom_point() + theme_minimal()' -o v.pdf