root/README

***MAKER Documentation***

#----------
INSTALLATION INSTUCTIONS FOR MAKER

*Step by step instructions are also available in the INSTALL text file.

To install maker, you will first need to install the following external programs:

     *PERL 5.8.0 or higher
     *BioPerl 1.5 or higher (www.bioperl.org)
     *Wu-BLAST 2.0  or higher (blast.wustl.edu)
     *SNAP version 2006-07-28  or higher (homepage.mac.com/iankorf)
     *RepeatMasker 3.1.6  or higher (www.repeatmasker.org)
     *Exonerate 1.4  or higher (www.ebi.ac.uk/~guy/exonerate)


You might want to also install these optional external programs:

     *Augustus 2.0  or higher (augustus.gobics.de)


You will also need to install the following perl modules if not already installed:

     *URI:Escape (search.cpan.org)
     *Error (search.cpan.org)
     *threads 1.67 or higher (search.cpan.org)
     *Datastore (http://www.yandell-lab.org/software/cgl.html)



To install mpi_maker, you must have an mpi package installed, try the following:

     *MPICH2 (http://www.mcs.anl.gov/research/projects/mpich2/)

note:  Remember to install MPICH2 with the --enable-sharedlibs flag set to the appropriate value (See MPICH2 Installer's Guide at http://www.mcs.anl.gov/research/projects/mpich2/documentation/index.php?s=docs).


Notes: 
1) RepeatMasker requires Wu-BLAST and a single file executable called TRF (see RepeatMasker website for details), so please install these before installing RepeatMasker
2) Exonerate Binaries can be downloaded from the website.  If you have Mac OSX, however, binaries are only available for version 1.0.  This verion will work too.  If you would like to compile exonerate, it requires GLIB, a C-library, that has a link from the exonerate website.  If you have Mac OSX, this can downloaded using FINK.
3) RepeatMasker requires a repeat library file, which is downloaded from Repbase (http://www.girinst.org/), this is explained on the RepeatMasker website.
4) Please note the location of all of the programs that you have installed.  You will need this information in the maker.exe file, one of MAKER's 3 control files.


Now that you have all the necessary programs installed, MAKER can be unpacked using:

tar xvfz maker.tar.gz

This will create a directory called maker with 5 sub directories:

        bin - contains the maker code.
        lib - contains all the necessary perl libaries for MAKER.
        MPI - contains MPI specific data to configure maker to run on a cluster that supports MPI.
        Apollo - contains gff3.tiers file (See section titled APOLLO below)
        data  - contains some sample data used to make sure everything works

Maker uses control files to guide each run.  Generic control files can be built using the -CTL flag in maker.  These control files can then be edited by the user to identify the location of all required input data and statistics.  Control files are run specific and seperate control will need to be built for each genome given to maker.  Maker will look for control files in the current working directory, so it is recomended that maker should be ran in a seperate directory containing unique control files for each genome.

Control files:

         1. maker_exe.ctl - contains the path information for needed executables
         2. maker_bopts - contains filtering statistics for BLAST and Exonerate
         3. maker_opts.ctl - contains all other information for MAKER, including the location of the input genome file.


Always remember to be examine the control files before each run of MAKER on your specific data


Programs required by maker rely on certain environmental variables being set.  If you have not set these variables per the installation instructions of the external programs, a reminder list is provided below:

for tcsh:
setenv PERL5LIB where_bioperl_is_installed
setenv WUBLASTMAT where_wublast_is_installed/matrix
setenv ZOE where_snap_is_installed
setenv WUBLASTFILTER where_wublast_is_installed/filter
setenv AUGUSTUS_CONFIG_PATH where_augustus_is_installed/config

for bash:
export PERL5LIB=where_bioperl_is_installed
export WUBLASTMAT=where_wublast_is_installed/matrix
export ZOE=where_snap_is_installed
export WUBLASTFILTER=where_wublast_is_installed/filter
export AUGUSTUS_CONFIG_PATH=where_augustus_is_installed/config


#----------
MPI MAKER INSTALL

If you are running maker on an MPI capable cluster, you can install an MPI version of maker by doing the following:

        1. Install standard maker and verify that it runs.
        2. Use cd to change to the MPI subdirectory in the maker instalation folder (i.e. maker/MPI/)
        3. Run Makefile.PL by typing:    perl Makefile.PL
        4. Compile needed binaries by typing:   make
        5. Install mpi_maker and all needed libraries by typing:        make install    

A new version of maker called mpi_maker should now be installed under maker/bin.

To run mpi_maker, first verify that your mpi environment is initiated, (i.e. using the mpdboot command). Now start mpi_maker via mpiexec.

Example:

        mpiexec -n 3 perl maker_directory/maker/bin/mpi_maker maker_opts.ctl maker_bopts.ctl maker_exe.ctl


Please see the documentation for the MPI environment you use for how to initiate an MPI process.


#----------
MAKER USAGE STATEMENT

Usage:

        maker [options] <maker_opts.ctl> <maker_bopts.ctl> <maker_exe.ctl>

        The three input arguments are user control files that specify how maker should behave.
        All input files listed in the control options files must be in fasta format.  Please
        see maker documentation to learn more about control file format.  The program will
        automatically try and locate the user control files in the current working
        directory if these arguments are not supplied when initializing maker.

        It is important to note that maker does not try and recalculated data that it has
        already calculated.  For example, if you run an analysis twice on the same fasta file
        you will notice that maker does not rerun any of the blast analyses but instead uses
        the blast analyses stored from the previous run.  To force maker to rerun all
        analyses, use the -f flag.

Options:

     -genome|g  <file_name>   Give MAKER a different genome file (this overrides the
                              control file value)

     -predictor <snap>        Selects the gene predictor to use when building annotations (Default
                <augustus>    is 'snap').  The option 'est2genome' builds annotations directly
                <est2genome>  from the EST evidence.

     -GFF                     Use an input gff3 format file of repeat elements for repeat masking.
                              You must set rm_gff in maker_opts.ctl to the files location.  This
                              option turns off all other repeat masking.

     -RM_off|R                Turns repeat masking off (* See Warning)

     -force|f                 Forces maker to rerun all analyses (replaces all previous output).

     -datastore|d             Causes output to be written using datastore.  This option is
                              automatically enabled if there are more than 1000 fasta entries
                              in the input file.  Output can then accessed using the
                              master_datastore_index file created by the program.

     -PREDS                   Outputs ab-initio predictions that do not overlap maker annotation
                              as gene annotations in the final gff3 output file (based on the
                              -predictor flag ).

     -CTL                     Generates generic control files in the current working directory.

     -retry     <integer>     Re-run failed contigs up to the specified number of re-tries.

     -cpus|c    <integer>     Tells how many cpus to use for Blast analysis (this overrides
                              contorol file value).

     -help|?                  Prints this usage statement.


Warning:
      
        *When using the -R flag, maker expects that the input genome file is already masked.
         Also if your genome file contains lower case characters, maker will consider those
         characers to be soft masked.


#----------
RUNNING MAKER WITH EXAMPLE DATA

1) Copy the files in the data directories to a temporary directory where you will run an example file.
2) Type maker -CTL to generate generic maker control files
3) Next you will need to edit the control files to include the path of the genome file, EST file, and proitein file, as well as the paths to all required executables.  See CONFIG FILE EDITING for more information.
4) Then try the following command from your temporary directory:

perl maker_directory/bin/maker maker_exe.ctl maker_opts.ctl maker_bopts.ctl

MAKER will create at least the following files/directory:

seq_name.gff - a gff file that can be loaded into GMOD, GBROWSE, or Apollo

seq_name.maker.transcripts.fasta - a file of the maker transcript sequences
seq_name.maker.snap.transcript.fasta - a file of ab-inito snap transcript sequences
seq_name.maker.proteins.fasta - a file of the maker protein sequences
seq_name.maker.snap.proteins.fasta - a file of ab-inito snap protein sequences

theVoid.seq_name - a directory containing all of the results files produced by maker, including BLAST reports, SNAP output, exonnerate output and the masked sequence

WARNING:
*The names of output files are based on sequence ids.  If giving maker a multi-fasta file, it is important to verify that all sequence id are unique, so files are not overwritten.
*If there are more than 1,000 sequences in a multi-fasta file or you use the -d flag on the command line a datastore structure will be used. see DATASTORE in this document.
*If sequence ids contain characters that are illegal in file names, those characters will be replaced automatically before building output file names.



#----------
DATASTORE

"Many filesystems have performance problems with large numbers of subdirectories and files within a single directory and even when the underlying filesystems handle things gracefully, access via network filesystems can be an issue.  The Datastore modules create a hiearchy of subdirectory layers, starting from a 'base', and mapping end-user's identifiers to the corresponding subdirectory." - quote from http://www.yandell-lab.org/  (See site for more information on the Datastore module)

Datastore will be used by maker if there are more than 1,000 sequences in a multi-fasta file or you use the -d flag on the command line.

When datastore is implemented, the output files described above will not appear where you would normally expect them to be.  Instead they will be located in a series of sub-directory under a new base-directory whose name is determined from the input genome file name, i.e. current_working_directory/input_genome_datastore/EE/Af/seq_name/seq_name.gff.  A master_datastore_index file will be made in the current working directory to help you find the output files from each sequence.

The master_datastore_index file is a file created to allow the user to easily find the exact output directory corresponding to contigs from the input genome file.  The The master_datastore_index file contains two columns of text; the first column shows the sequence identifier from each fasta header, and the second column shows the location of the output files for that sequence. 



#----------
CONFIG FILE EDITING

Lines in the maker control files have the format key:value whith no spaces before or after the colon(:).  If the value is a file name, you can use relative paths and environmental variables, i.e. genome:$HOME/my_genome.fasta


MAKER has 3 control files for configuration options.

A. maker_exe.ctl - includes information about programs executed by MAKER.

Here is what the standard maker_exe.ctl control file looks like:
====================================

#-----Location of executables required by Maker
xdformat:/usr/local/wublast/xdformat              #location of xdformat executable
blastn:/usr/local/wublast/blastn                  #location of blastn executable
blastx:/usr/local/wublast/blastx                  #location of blastn executable
snap:/usr/local/snap/snap                         #location of snap executable
augustus:/usr/local/augustus/bin/augustus         #location of augustus executable (optional)
RepeatMasker:/usr/local/RepeatMasker/RepeatMasker #location of RepeatMasker executable
exonerate:/usr/local/exonerate/bin/exonerate      #location of exonerate executable

====================================

Note that for all control files the comments written to help users begin with a pound sign(#).  In addition, options before the colon(:) can not be changed, nor should there be a space before or after the colon.


B. maker_bopts.ctl - contains statistics for fltering blast and exonerate data

Here an example maker_bopts.ctl:
====================================

#-----BLAST and Exonerate statistics thresholds
percov_blastn:0.80 #Blastn Percent Coverage Threhold EST-Genome Alignments
percid_blastn:0.85 #Blastn Percent Identity Threshold EST-Genome Aligments
eval_blastn:1e-10  #Blastn eval cutoff
bit_blastn:40      #Blastn bit cutoff
percov_blastx:0.50 #Blastx Percent Coverage Threhold Protein-Genome Alignments
percid_blastx:0.40 #Blastx Percent Identity Threshold Protein-Genome Aligments
eval_blastx:1e-6   #Blastx eval cutoff
bit_blastx:30      #Blastx bit cutoff
e_perc_cov:50      #Exonerate Percent Coverage Thresshold EST_Genome Alignments
ep_score_limit:20  #Report  alignments scoring at least this percentage of the maximal score exonerate nucleotide
en_score_limit:20  #Report  alignments scoring at least this percentage of the maximal score exonerate protein

====================================


C. maker_opts.ctl - contains options for maker and external programs used by maker

Here an example maker_opts.ctl:
====================================

#-----sequence and library files
genome:fly_assembly.fasta        #genome sequence file (required)
est:fly_est.fasta                #EST sequence file (required)
protein:uniprot.fasta            #protein sequence file (required)
repeat_protein:te_proteins.fasta #a database of transposable element proteins
rmlib:fly_specific_repeats.fasta #an organism specific repeat library (optional)
rm_gff:                          #a gff3 format file of repeat elements (only used with -GFF flag)

#-----external application specific options
snaphmm:fly          #SNAP HMM model
augustus_species:fly #Augustus gene prediction model
model_org:all        #RepeatMasker model organism
alt_peptide:c        #amino acid used to replace non standard amino acids in xdformat
cpus:2               #max number of cpus to use in BLAST and RepeatMasker

#-----Maker specific options
predictor:snap     #identifies which gene prediction program to use for annotations
te_remove:1        #mask regions with excess similarity to transposable element proteins
max_dna_len:100000 #length for dividing up contigs into chunks (larger values increase memory usage)
split_hit:10000    #length of the splitting of hits (max intron size for EST and protein alignments)
snap_flank:200     #length of sequence surrounding EST and protein evidence used to extend gene predictions
single_exon:0      #consider EST hits aligning to single exons when generating annotations, 1 = yes, 0 = no
use_seq_dir:1      #place output files in same directory as sequence file: 1 = yes, 0 = no
clean_up:0         #remove theVoid directory: 1 = yes, 0 = no

====================================


#----------
ADDING UTRs for GBROWSE

* When using APOLLO to visualize gene annotations, UTRs are inferred based on exon and CDS locations.  However GMOD and GBROWSE do not infer the UTR, so to visualize the UTR, you will have to run: add_utr_gff.pl with the following command:

maker2zff.pl <directory>
<directory> is the directory where all of your GFF files are located

each GFF file will have a sister file called sequence.wutr.gff3


#----------
APOLLO

Maker is bundled with a configuration file that improves the color and display of maker annotations and evidence in the Apollo genome browser.  The configuration file is called "gff3.tiers" and is located in the maker/Apollo/ directory.  The file should be copied to the conf/ sub_directory which is located under the Apollo instalation directory.  Using the Mac version of Apollo the conf/ directory is located at /Applications/Apollo.app/Contents/Resources/app/conf/.


#----------
HMM BUILDING (based snap documentation)

A.  First you will need to determine the genes used to model future genes, by determining a high quality gene set (annotations for the high quality gene should be in GFF3 format).  The high quality gene set can then be coverted into snap ZFF format using maker2zff.pl found in maker/bin.

This program is run with the following command:

      maker2zff.pl <directory> genome

*<directory> is the directory where all of your GFF3 files are located
*geneome is the name for the outfile

Files Created:

      genome.ann
      genome.dna

Note:  A convenient way to identify and initial high quality gene set for the HMM is to use the -predictor est2genome option in maker.  This will produce gene annotations based solely on EST evidence.  These annoations can then seed the first HMM.  After running maker again using this new HMM and the -predictor snap option, you can use the second round of annotations as the seed for an even better HMM model.  In this way the HMM model progressively improves with each run of maker.

Another strategy for identifying an initial gene set to model the HMM is to use the program CEGMA (http://korflab.ucdavis.edu/software.html).  CEGMA builds a highly reliable set of gene annotations in the absence of experimental data by identifying DNA regions with homology to a set of 458 proteins that are highly conserved among taxa.

Combining both CEGMA and maker datasets to build the first HMM is also a good strategy.


B.  Next you will use the dna and zff file (genome.dna and genome.ann) to produce a SNAP HMM as described in the SNAP documation (which we have provided below):

The first step is to look at some features of the genes:

    fathom genome.ann genome.dna -gene-stats 

Next, you want to verify that the genes have no obvious errors:

    fathom genome.ann genome.dna -validate

You may find some errors and warnings. Check these out in some kind of genome
browser and remove those that are real errors. Next, break up the sequences into
fragments with one gene per sequence with the following command:

    fathom -genome.ann genome.dna -categorize 1000

There will be up to 1000 bp on either side of the genes. You will find
several new files.

    alt.ann, alt.dna (genes with alternative splicing)
    err.ann, err.dna (genes that have errors)
    olp.ann, olp.dna (genes that overlap other genes)
    wrn.ann, wrn.dna (genes with warnings)
    uni.ann, uni.dna (single gene per sequence)

Convert the uni genes to plus stranded with the command:

    fathom uni.ann uni.dna -export 1000 -plus

You will find 4 new files:

    export.aa   proteins corresponding to each gene
    export.ann  gene structure on the plus strand
    export.dna  DNA of the plus strand
    export.tx   transcripts for each gene

The parameter estimation program, forge, creates a lot of files. You probably
want to create a directory to keep things tidy before you execute the program.

    mkdir params
    cd params
    forge ../export.ann ../export.dna
    cd ..

Last is to build an HMM.

    hmm-assembler.pl my-genome params > my-genome.hmm


Lastly, you will want to add the location of your hmm file to your maker_opts.ctl file.

*For more information see SNAP documentation on how to build an HMM