Lecture #23 looked at planning under uncertainty, and, in particular, at Markov Decision Processes.
Notes are available in four formats:
The material from this lecture isn't covered in any detail in the textbook. However, there are two very good papers which cover this and related material. These are:
C. Boutilier, T. Dean and S. Hanks, Decision-theoretic planning: structural assumptions and computational leverage, Journal of Artificial Intelligence Research, 11, 1-94, 1999.and
L. P. Kaelbling, M. L. Littman and A. R. Cassandra, Planning and acting in partially observable stochastic domains, Artificial Intelligence, 101, 99-134, 1998.