library(readsent)

read_until(?Delimiters, -Answer)

reads characters from the current input until a character in the Delimiters string is read. The characters are accumulated in the Answer string, and include the closing delimiter. The end-of-file character always acts as a delimiter, even if it is not in the list of characters you supply.

trim_blanks(+RawInput, ?Cleaned)

removes leading and trailing layout characters from RawInput, and replaces internal groups of layout characters by single spaces.

chars_to_words(+Chars, ?Words)

parses a list of characters (read by read_until) into a list of tokens, where a token is

X

X a full stop or other punctuation mark

atom(X)

X a sequence of letters, e.g. atom(the)

integer(X)

X a sequence of digits, e.g. integer(12)

apost

'

aposts

's

string(X)

X "..sequence of any.."

Thus the string the "Z-80" is on card 12. would be parsed as [atom(the),string('Z-80'),atom(is),atom(on),atom(card), integer(12),'.']. It is up to the sentence parser to decide what to do with these. Note that the final full stop, if any, is retained, as the parser may need it.

case_shift(+Mixed, ?Lower)

converts all the upper case letters in Mixed to lower case. Other characters (not necessarily letters!) are left alone. If you decide to accept other characters in words only chars_to_atom has to alter. See also lower/2 in library(caseconv).

read_line(-Chars)

reads characters up to the next newline or the end of the file, and returns them in a list, including the newline or end-of-file. Usually you want multiple spaces conflated to one, and the newline dropped. To do this, call trim_blanks on the result. For a routine that does not include the newline character in the result, see the predicate get_line/1 in library(lineio).

read_sent(-Sentence)

reads a single sentence from the current input stream. It reads characters up to the first sentence terminator (as defined by is_period/1) it finds, then throws characters away until it has reached the end of a line. The characters read are then broken up into "words", where a "word" is

• a sequence of letters, which is converted to lowercase and returned as a compound term atom(Word). For example, THIS would be returned as atom(this)
• a sequence of decimal digits, which is returned as a compound term integer(Value). For example, 0123 would be returned as integer(123). Plus and minus signs become separate atoms.
• a sequence of characters in double quotes, which is returned as a compound term string(X), where X is an atom containing the characters between the quotes. Two adjacent quotes are read as one, so the input string Double "" Quote is returned as string('Double " Quote').
• apostrophe s ('s) is returned as the atom aposts.
• apostrophe not followed by s (') is returned as the atom apost.
• any other graphic character is returned as a single-character Prolog atom.

The resulting string is returned in Sentence. Here is an example.

          | ?- read_sent(X).
          |: The predicate read_sent/1 accepts sentences
          |: that span more than 1 line, but not lines
          |: that contain +2 or more sentences.  trash trash
          
          X = [atom(the),atom(predicate),string(read_sent),
               atom(accepts),atom(sentences),atom(that),
               atom(span),atom(more),atom(than),integer(1),
               atom(line),',',atom(but),atom(not),atom(lines),
               atom(that),atom(contain),+,integer(2),atom(or),
               atom(more),atom(sentences),.]
          

This is more unwieldy than the output of read_in/1, but it does mean that your parser can tell the difference between words, numbers, and strings by pattern matching rather than having to use the meta-logical predicates atom/1, integer/1, and so forth.