Introduction to theory of computation

Tom Carter

http://cogs.csustan.edu/˜ tom/SFI-CSSS

Complex Systems Summer School

June, 2002

Our general topics: ←

⊚ Symbols, strings and languages ⊚ Finite automata ⊚ Regular expressions and languages ⊚ Markov models ⊚ Context free grammars and languages ⊚ Language generators and recognizers ⊚ The Chomsky hierarchy ⊚ Turing machines ⊚ Computability and tractability ⊚ Computational complexity ⊚ References

Introduction ←

What follows is an extremely abbreviated look at some of the important ideas of the general areas of automata theory, computability, and formal languages. In various respects, this can be thought of as the elementary foundations of much of computer science. It also includes a wide variety of tools, and general categories of tools …

Symbols, strings and languages ←

• The classical theory of computation traditionally deals with processing an input string of symbols into an output string of symbols. Note that in the special case where the set of possible output strings is just {‘yes’, ‘no’}, (often abbreviated {T, F} or {1, 0}), then we can think of the string processing as string (pattern) recognition.

We should start with a few definitions. The first step is to avoid defining the term ‘symbol’ – this leaves an open slot to connect the abstract theory to the world …

We define:

1. An alphabet is a finite set of symbols.

2. A string over an alphabet A is a finite ordered sequence of symbols from A. Note that repetitions are allowed. The length of a string is the number of symbols in the string, with repetitions counted. (e.g., |aabbcc | = 6)

3. The empty string, denoted by ϵ, is the (unique) string of length zero. Note that the empty string ϵ is not the same as the empty set ∅.

4. If S and T are sets of strings, then ST = {xy| x ∈ S and y ∈ T}

5. Given an alphabet A, we define

A^{0} = {ϵ}
A^{n+1} = AA^{n}
A^{∗} = ∞ ; ⋃ ; n=0 A^{n}

6. A language L over an alphabet A is a subset of A^{∗}. That is, L ⊂ A^{∗}.

• We can define the natural numbers, ℕ, as follows:

We let

0 = ∅
1 = {∅}
2 = {∅, {∅}}
and in general
n + 1 = {0, 1, 2, …, n}.

Then

ℕ = {0, 1, 2, …}.

• Sizes of sets and countability:

1. Given two sets S and T, we say that they are the same size (|S| = |T|) if there is a one-to-one onto function f : S → T.

2. We write |S| ≤ |T| if there is a one-to-one (not necessarily onto) function f : S → T.

3. We write |S| < |T| if there is a one-to-one function f : S → T, but there does not exist any such onto function.

4. We call a set S

(a) Finite if |S| < |ℕ|

(b) Countable if |S| ≤ |ℕ|

(c) Countably infinite if |S| = |ℕ|

(d) Uncountable if |ℕ| < |S|.

5. Some examples:

(a) The set of integers ℤ = {0, 1, −1, 2, −2, …} is countable.

(b) The set of rational numbers ℚ = {p/q | p, q ∈ ℤ, q ≠ 0} is countable.

(c) If S is countable, then so is SxS , the cartesian product of S with itself, and in general so is S^{n} for any n < ∞.

(d) For any nonempty alphabet A, A^{∗} is countably infinite.

Exercise: Verify each of these statements.

6. Recall that the power set of a set S is the set of all subsets of S:

P(S) = {T | T ⊂ S}.

We then have the fact that for any set S,

|S| < |P(S)|.

Pf: First, it is easy to see that

|S| ≤ |P(S)|

since there is the one-to-one function f : S → P(S) given by f(s) = {s} for s ∈ S.

On the other hand, no function f : S → P(S) can be onto. To show this, we need to exhibit an element of P(S) that is not in the image of f. For any given f, such an element (which must be a subset of S) is

R_{f} = {x ∈ S | x ∉ f(x)}.

Now suppose, for contradiction, that there is some s ∈ S with f(s) = R_{f}. There are then two possibilities: either s ∈ f(s) = R_{f} or s ∉ f(s) = R_{f}. Each of these lead to a contradiction: If s ∈ f(s) = R_{f}, then by the definition of R_{f}, s ∉ f(s). This is a contradiction. If s ∉ f(s) = R_{f}, then by the definition of R_{f}, s ∈ R_{f} = f(s). Again, a contradiction. Since each case leads to a contradiction, no such s can exist, and hence f is not onto. QED • From this, we can conclude for any

countably infinite set S, P(S) is uncountable. Thus, for example, P(ℕ) is uncountable. It is not hard to see that the set of real numbers, ℝ, is the same size as P(ℕ), and is therefore uncountable.

Exercise: Show this. (Hint: show that ℝ is the same size as (0, 1) = {x ∈ ℝ | 0 < x < 1}, and then use the binary representation of real numbers to show that |P(ℕ)| = |(0, 1)|.

• We can also derive a fundamental (non)computability fact:

There are languages that cannot be recognized by any computation. In other words, there are languages for which there cannot exist any computer algorithm to determine whether an arbitrary string is in the language or not.

To see this, we will take as given that any computer algorithm can be expressed as a computer program, and hence, in particular, can be expressed as a finite string of ascii characters. Therefore, since ASCII^{∗} is countably infinite, there are at most countably many computer algorithms/programs. On the other hand, since a language is any arbitrary subset of A^{∗} for some alphabet A, there are uncountably many languages, since there are uncountably many subsets.

Finite automata ←

• This will be a quick tour through some of the basics of the abstract theory of computation. We will start with a relatively straightforward class of machines and languages – deterministic finite automata and regular languages.

In this context when we talk about a machine, we mean an abstract rather than a physical machine, and in general will think in terms of a computer algorithm that could be implemented in a physical machine. Our descriptions of machines will be abstract, but are intended to be sufficiently precise that an implementation could be developed.

• A deterministic finite automaton (DFA) M = (S, A, s_{0}, δ, F) consists of the following:

S, a finite set of states,

A, an alphabet,

s_{0} ∈ S, the start state,

δ : SxA → S, the transition function, and

F ⊂ S, the set of final (or accepting) states of the machine.

We think in terms of feeding strings from A^{∗} into the machine. To do this, we extend the transition function to a function

δ̂ : SxA^{∗} → S

by

δ̂(s, ϵ) = s,
δ̂(s, xa) = δ̂(δ(s, a), x).

We can then define the language of the machine by

L(M) = {x ∈ A^{∗} | δ̂(s_{0}, x) ∈ F}.

In other words, L(M) is the set of all strings in A^{∗} that move the machine via its transition function from the start state s_{0} into one of the final (accepting) states.

We can think of the machine M as a recognizer for L(M), or as a string processing function

f_{M} : A^{∗} → {1, 0}

where f_{M}(x) = 1 exactly when x ∈ L(M).

• There are several generalizations of DFAs that are useful in various contexts. A first important generalization is to add a nondeterministic capability to the machines. A nondeterministic finite automaton (NFA) M = (S, A, s_{0}, δ, F) is the same as a DFA except for the transition function:

S, a finite set of states, A, an alphabet, s_{0} ∈ S, the start state, δ : SxA → P(S), the transition function, F ⊂ S, the set of final (or accepting)

states of the machine. For a given input symbol, the transition function can take us to any one of a set of states. We extend the transition function to δ̂ : SxA^{∗} → P(S) in much the same way:

δ̂(s, ϵ) = s,
δ̂(s, xa) = _{r∈δ(s,a)}^{⋃} δ̂(r, x).

We define the language of the machine by L(M) = {x ∈ A^{∗} | δ̂(s_{0}, x) ∩ F ≠ ∅}.

A useful fact is that DFAs and NFAs define the same class of languages. In particular, given a language L, we have that L = L(M) for some DFA M if and only if L = L(M′) for some NFA M′.

Exercise: Prove this fact.

In doing the proof, you will notice that if L = L(M) = L(M′) for some DFA M and NFA M′, and M′ has n states, then M might need to have as many as 2^{n} states. In general, NFAs are relatively easy to write down, but DFAs can be directly implemented.

• Another useful generalization is to allow the machine to change states without any input (often called ϵ-moves). An NFA with ϵ-moves would be defined similarly to an NFA, but with transition function

δ : Sx(A ∪ {ϵ}) → P(S).