Required format of each dataset:
Each dataset file must be a tab-delimited matrix: the first row should be condition names, the first column should be gene names, and the matrix entries should be expression values. For simplicity, genes must be in Entrez ID's (we select Entrez because it is the gene naming system widely mapped across different human platforms). Datasets may originate from any of the gene expression hybridization platforms or sequencing platforms, though in the latter case users should collapse read abundances into gene-based values. It is expected that the conditions within each dataset should be comparable in their expression - this can be achieved by standard normalization procedure such as quantile normalization, variance-stabilizing normalization. Generally speaking, expressions need not be comparable across datasets as SEEK will examine the correlation structures within each dataset and appropriately normalize at the level of correlations.

Though each dataset must have >3 conditions to make possible Pearson correlation computation, having more conditions per dataset leads to better search performance. More conditions can be introduced by replicates or having a diverse set of conditions in each dataset. For datasets with fold-change differences, these are also acceptable inputs so long that each condition's expression fold-change values follow an approximately normal distribution. Expression values must be log-normalized, or log(1+counts) normalized in the case of sequencing data.

Create a dataset list:
Once datasets have been prepared, next prepare a dataset description file, which defines datasets. This is a 3-column tab-delimited file with columns as: file_name, dataset_name, platform_name. For your reference, below is a sample dataset description. The platform column is a GEO specific code indicating the experimental technology. In this case, this is GPL570, the popular Affymetrix HGU133-Plus 2 microarray platform. It can also be any value that the user defines.

GSE13494.GPL570.pcl	GSE13494.GPL570	GPL570
GSE17907.GPL570.pcl	GSE17907.GPL570	GPL570
GSE45584.GPL6480.pcl	GSE45584.GPL6480	GPL6480

Examples
A collection of datasets: breast_cancer_dset.tar.gz
The corresponding dataset.description.txt file: dataset.description.txt

Creating the compendium entails several tedious but necessary steps, including: calculating the correlation matrix, normalizing correlations, and data organizations for efficiency purposes. In order to make the process of creating the compendium easier, we provide below a set of scripts which automate these tasks.
Download: scripts.tar.gz

Please ensure that you have enough space. A 10-dataset compendium needs 12GB for building it.

Extract: tar -zxf scripts.tar.gz
Next, run the prepare_seek.py in the scripts:

./prepare_seek.py <dataset directory> 
   <dataset list> 
   <setting directory> 
   <path to sleipnir binary> 
   <output directory>

where dataset directory contains the dataset matrix files; dataset list is the dataset listing; setting directory contains the setting files (gene_map.txt and quant2); path to sleipnir is where sleipnir is installed to; output directory is where compendium will be created

Example workflow:

mkdir test.seek
cd test.seek
tar -zxf ~/Downloads/breast_cancer_dset.tar.gz
cp ~/Downloads/dataset.description.txt .
~/scripts/prepare_seek.py pcl dataset.description.txt 
   ~/scripts ~/sleipnir_build/bin .

What this script will do:

• Calculate correlation matrix
• Calculate gene z-score average for gene hubbiness correction
• Join correlation matrices followed by splitting correlation vectors by genes to faciliate searching
• Calculate platform-wide gene z-score average for normalization

A number of decisions have been made by the script, and which should work best for most encountered situations. These decisions are next described. For the correlation measure, SEEK chooses the Pearson correlation coefficient followed by Fisher's transform and then a z-score normalization of the resulting normal distribution. It is found that Pearson works well in most datasets that have been properly normalized (RMA or MAS5). It has a very comparable performance to Spearman correlation. SEEK bounds all correlation z-score values within the range [-5, 5] in order to reduce the effect of extreme z-score values. During the gene-hubbiness computation stage, SEEK calculates an average connectivity score or average z-score for each gene to correct for hubby genes (these can have the undesirable effect of overwhelming the coexpressed gene ranking).

The entire process of completing the set of tasks in the script will take ~3 hours/10-dataset, with 5GB of RAM needed.

Please note that the bottleneck is in the Data2DB step (joining and splitting correlation matrices). Memory usage is quite heavy. But it can be adjusted. For a large compendium with 100-300 datasets, the current setting in the script will process 50 datasets at a time, consuming 25GB of physical memory in the process. If you do not have enough memory, adjust 50 to some lower number (eg. 20) (see -B parameter in Data2DB line).

Afterward, you will have set up SEEK compendium. You will find the following new directories created: sinfo, plat, prep, dab, db, pclbin.