Iterators with Complex State

	Programming in Lua
	Part I. The Language              Chapter 7. Iterators and the Generic for

---

7.4 - Iterators with Complex State

Frequently, an iterator needs to keep more state than fits into
a single invariant state and a control variable.
The simplest solution is to use closures.
An alternative solution is to pack all it needs into a table
and use this table as the invariant state for the iteration.
Using a table, an iterator can keep as much data as it needs
along the loop.
Moreover, it can change that data as it goes.
Although the state is always the same table
(and therefore invariant),
the table contents change along the loop.
Because such iterators have all their data in the state,
they typically discard the second argument
provided by the generic for
(the iterator variable).

As an example of this technique,
we will rewrite the iterator allwords,
which traverses all the words from the current input file.
This time, we will keep its state using a table with two fields,
line and pos.

The function that starts the iteration is simple.
It must return the iterator function and the initial state:

    local iterator   -- to be defined later
    
    function allwords ()
      local state = {line = io.read(), pos = 1}
      return iterator, state
    end

The iterator function does the real work:

    function iterator (state)
      while state.line do        -- repeat while there are lines
        -- search for next word
        local s, e = string.find(state.line, "%w+", state.pos)
        if s then                -- found a word?
          -- update next position (after this word)
          state.pos = e + 1
          return string.sub(state.line, s, e)
        else    -- word not found
          state.line = io.read() -- try next line...
          state.pos = 1          -- ... from first position
        end
      end
      return nil                 -- no more lines: end loop
    end

Whenever it is possible,
you should try to write stateless iterators,
those that keep all their state in the for variables.
With them, you do not create new objects when you start a loop.
If you cannot fit your iteration into that model,
then you should try closures.
Besides being more elegant,
typically a closure is more efficient
than an iterator using tables:
First, it is cheaper to create a closure than a table;
second, access to upvalues is faster than
access to table fields.
Later we will see yet another way to write iterators,
with coroutines.
This is the most powerful solution,
but a little more expensive.

---

Programming in Lua