Subject: Re: reminder

From: "Kurt A. Shoens" <[email-masked]>
Date: Thu, 01 Jul 93 10:39:26 -0800

Hector ... since we're going to talk about property extraction, this
might be a good time to send along that classifier stuff I promised
long long ago.

I'll just put them all together in a long message.  The first thing is
the "schema" file, which describes the data types the classifier is
looking for and the regular expression based features.  The keyword
"extract" means that there is a C function named "ex_DATATYPE" that
finds other features.  "firstpats" are regular expressions that must
appear in the first line.  "restpats" can occur anywhere.  "filename"
gives regular expression patterns on the file name.  | in the patterns
below means alternation.

There are currently 17 programmed features:

Binary_isbinary                 has unprintable characters
CProgram_dirmake                directory has a Makefile/makefile
CProgram_dotc                   directory has .c files
CProgram_doth                   directory has .h files
CProgram_manpage                directory has man page nroff source
Compress_magic                  file starts with the compressed file magic #
Dir_isdir                       file is a directory
File_fileopen                   file is readable
MHFolder_numbered               directory contains numbered files
MHFolder_rfc822s                dir contains files class as RFC822
MHFolder_unnumbered             dir contains unnumbered files
Symlink_islink                  file is a symbolic link
Tar_Blockandcheck               first block correctly cksums as tar file
Text_istext                     file contains printable characters
Uuencode_terminator             file end looks like end of uuencoded file
exe6000_magic                   file has magic # to be AIX/6000 a.out
obj6000_magic                   file has magic # to be AIX/6000 .o file

I was going to put the output of the training process in this same
message, but it's huge, so I'll send it separately.

File {
        extract
}

Text {
        extract
}

Binary {
        extract
}

Directory {
        extract
}

RFC822 {
        firstpats {
                "^Received:|^Delivery-Date:" regexp must
        }
        restpats {
                "^From:" regexp
                "^To:" regexp
                "^Message-Id:" regexp
        }
}

Usenet {
        firstpats {
                "^Path:" regexp must
        }
        restpats {
                "^Message-ID:" regexp
                "^Newsgroups:" regexp
                "^Lines:" regexp
        }
}

Mbox {
        firstpats {
                "^From " regexp must
        }
        restpats {
                "^From:" regexp
                "^To:" regexp
                "^Message-Id:" regexp
        }
}

Compress {
        extract
        filenames {
                ".Z$|.z$" regexp
        }
}

Tar {
        extract
}

Lib6000 {
        filenames {
                "\\.a$" regexp
        }
        firstpats {
                "<aiaff>" must
        }
}

Obj6000 {
        extract
        filenames {
                "\\.o$" regexp
        }
}

Exe6000 {
        extract
}

PostScript {
        firstpats {
                "^%!" regexp must
        }
        restpats {
                "%EndComments"
                "currentmatrix"
                "newpath"
                "%Creator:"
                "showpage"
                "grestore"
                "EndProlog"
                "closepath"
                "setmatrix"
                "definefont"
                "itransform"
        }
}

CSource {
        filenames {
                "\\.c$" regexp must
        }
        restpats {
                "^#include" regexp
                "&#define" regexp
                "stdio"
                "struct"
                "printf"
                "NULL"
                "argv"
                "argc"
        }
}

Troffman {
        restpats {
                "^\\.TH" regexp
                "^\\.IR" regexp
                "^\\.SH *SEE ALSO" regexp
                "^\\.SH *AUTHOR" regexp
                "^\\.SH *SYNOPSIS" regexp
                "^\\.SH *DESCRIPTION" regexp
        }
}

Troffme {
        restpats {
                "^\\.bp" regexp
                "^\\.de" regexp
                "^\\.he *'" regexp
                "^\\.fi" regexp
                "^\\.na" regexp
                "^\\.nr" regexp
                "^\\.po" regexp
                "^\\.pp" regexp
                "^\\.sz" regexp
        }
}

GIF {
        firstpats {
                "^GIF87a" regexp must
        }
}

Frame {
        firstpats {
                "^<MakerFile" regexp must
        }
}

PBM {
        firstpats {
                "^P[0-9] |^P[0-9]$" regexp must
        }
}

CHeader {
        filenames {
                "\\.h$" regexp
        }
}

Shellscript {
        firstpats {
                "^#! */bin/[a-z]*sh" regexp must
        }
        restpats {
                "alias"
                "cd"
                "elseif"
                "export"
                "foreach"
                "login"
                "logout"
                "nice"
                "popd"
                "pushd"
                "rehash"
                "set"
                "setenv"
                "source"
                "umask"
                "unalias"
                "unset"
                "unsetenv"
        }
}

Tele {
        restpats {
                "IBM Internal Use Only"
                "^\\.\\. *VERSION" regexp
                "^\\.\\. *Security" regexp
                "^\\.\\. *Data|^\\.\\. *DATA" regexp
                "^\\.\\. *Field|^\\.\\. *FIELD" regexp
                ":Details.Employee"
                ":Details.Department"
        }
}

PALS {
        restpats {
                "^NUMBER " regexp
                "^DATE " regexp
                "^CATAGORY " regexp
                "^TYPE " regexp
                "^TITLE " regexp
                "^ABSTRACT " regexp
        }
}

Script {
        restpats {
                "^\\.\\*" regexp
                "^\\.cm" regexp
                "^\\.pm" regexp
                "^\\.rf" regexp
                "^\\.rh" regexp
                "^\\.se" regexp
                ":address."
                ":aline."
                ":body."
                ":date."
                ":dd."
                ":dl."
                ":dt."
                ":eaddress."
                ":edl."
                ":ehp2."
                ":eol."
                ":eul."
                ":exmp."
                ":frontm."
                ":h1."
                ":h2."
                ":hp2."
                ":li."
                ":ol."
                ":p."
                ":q."
                ":ul."
        }
}

TeX {
        restpats {
                "\\bf"
                "\\def\\"
                "\\hbox{"
                "\\noindent"
        }
}

LaTeX {
        restpats {
                "\\end{"
                "\\begin{"
                "{document}"
                "{article}"
                "\\section{"
                "\\textwidth{"
        }
}

FoilTeX {
        restpats {
                "\\Restriction{"
                "{foils}"
                "\\itemize{"
                "\\foilhead{"
        }
}

BibTeX {
        restpats {
                "^@manual\\{" regexp
                "^@inproceedings\\{" regexp
                "^@article\\{" regexp
                "^@techreport\\{" regexp
                "^title *=" regexp
                "^author *=" regexp
                "^keywords *=" regexp
                "^readers *=" regexp
                "^random1 *=" regexp
        }
}

CMakefile {
        filenames {
                "/[Mm]akefile$|^[Mm]akefile$" regexp
        }
}

MHFolder {
        extract
}

CProgram {
        extract
}

Slate {
        firstpats {
                "^<list>" regexp must
        }
        restpats {
                "^<protocol bbn-multimedia-format>" regexp must
        }
}

Notebook {
        firstpats {
                "^==========" regexp
        }
}

Core {
        filenames {
                "^core|/core" regexp
        }
}

Xdefaults {
        filenames {
                "^\\.Xdefaults|/\\.Xdefaults" regexp
        }
        restpats {
                "^Mwm\\*" regexp
                ":[     ]*pointer" regexp
                "*fontList:"
                "*geometry:"
                "*keyboardFocusPolicy:"
                "*background:"
                "*foreground:"
        }
}

Newsrc {
        filenames {
                "^\\.newsrc|/\\.newsrc" regexp
        }
        restpats {
                "^[a-z]+(\\.[a-z]+)*[:!]( [0-9]+)|^[a-z]+(\\.[a-z]+)*[:!]$" regexp
        }
}

Mod3Inter {
        filenames {
                "\\.i3$" regexp must
        }
        restpats {
                "INTERFACE "
        }
}

Mod3Impl {
        filenames {
                "\\.m3$" regexp must
        }
        restpats {
                "^[     ]*IF " regexp
                "^[     ]*VAR |\\([     ]*VAR |;[       ]*VAR " regexp
                "^[     ]*BEGIN$" regexp
                "^[     ]*MODULE .*;$" regexp
                "RETURN .*;" regexp
                "^[     ]*IMPORT " regexp
        }
}

RnKill {
        filenames {
                "^KILL|/KILL" regexp must
        }
        firstpats {
                "^THRU" regexp must
        }
}

Bitmap {
        firstpats {
                "_width " must
        }
        restpats {
                "_height " must
                "_bits[] = {" must
        }
}

Lispemacs {
        restpats {
                "\\(interactive[ )]" regexp
                "\\(if *\\(" regexp
                "-to-"
                "-char "
                "save-"
                "(setq "
                "(defun "
        }
}

TeXsty {
        filenames {
                "\\.sty$" regexp must
        }
        restpats {
                "\\def\\|\\def "
        }
}

Uuencode {
        extract
        firstpats {
                "^begin [0-9][0-9][0-9] " regexp
        }
}

Yacc {
        filenames {
                "\\.y$" regexp
        }
        restpats {
                "%prec "
                "yyerror"
                "^%token[       ]" regexp
                "%type[         ]" regexp
                "%union |%union$" regexp
        }
}

Dvi {
        filenames {
                "\\.dvi$" regexp must
        }
        firstpats {
                "\xf7\x02\x01\x83\x92\xc0\x1c\x3b"
        }
}

Tif {
        firstpats {
                "^II\\*|^MM\\*" regexp must
        }
}