Teaching code

4S010675

Credits

12

Coordinator

Alberto Castellini

Language

English

Scientific Disciplinary Sector (SSD)

ING-INF/05 - INFORMATION PROCESSING SYSTEMS

Courses Single

Authorized

Moodle Seminars 0

The teaching is organized as follows:

Learning objectives

The reinforcement learning module introduces students to reinforcement learning and planning under uncertainty. In particular, it is focused on the design of algorithms that enable machines to learn based on reinforcements, hence from partial, implicit and delayed feedback obtained by repeatedly interact with the environment or users. At the end of the course, students will have to demonstrate that they have acquired the ability to i) tackle sequential decision problems with reinforcement learning techniques, ii) identify and apply the most effective and efficient algorithms to solve specific sequential decision problems, iii) designing new reinforcement learning algorithms. In particular, the acquired knowledge concerns advanced techniques for the resolution of Markov Decision Process (eg, research with Monte Carlo methods), bandit problems, model-based and model-free reinforcement learning, Bayesian reinforcement learning, deep reinforcement learning, and advanced reinforcement learning techniques (safe policy improvement, partially observable environments, hierarchical reinforcement learning, imitation-based learning, inverse reinforcement learning, and meta-learning). The Advanced Programming for AI module aims to provide information on programming languages, tools and software architectures that have emerged in the field of software systems (SW) based on artificial intelligence. The goal is to provide students with an understanding of the specific characteristics and key principles underlying various languages and advanced tools and to solve some classes of AI problems. Students will acquire programming skills in Python, will be able to write programs to solve typical problems and to assemble software modules, manage models, patterns, and perform the deployment of those modules on cloud platforms, with particular focus on interoperability and explainability.

Prerequisites and basic notions

Being a first year, first semester exam, there are no specific prerequisites other than those required for access to the degree course.

Program

------------------------
UL: Reinforcement Learning
------------------------
- Introduction to RL
- Multi-armed Bandits
- Markov Decision Processes
- RL based on Dynamic Programming (e.g., value/policy iteration)
- RL based on Monte Carlo Methods
- RL based on Temporal-Difference Learning (e.g., Q-learning, Sarsa)
- Planning and Learning: Model-Based RL (e.g., Dyna, MCTS planning)
- RL with Approximate Solutions (e.g., DQN)
- Policy Gradient Methods (e.g., Reinforce)
- RL with Actor-Critic Methods (e.g., A2C)
- Advanced algorithms (e.g., TRPO, PPO, SAC, DDPG)
- RL in Partially Observable Environments
- RL in Continuous Action Spaces
- Advanced reinforcement learning techniques
--- safe policy improvement
--- hierarchical reinforcement learning
--- imitation-based learning
--- inverse reinforcement learning
--- meta-learning
------------------------
UL: Advanced programming for AI
------------------------
The Advanced Programming for AI module aims to provide information on programming languages, tools, and software architectures that have emerged in the field of artificial intelligence. Students will advance their programming skills in Python, assemble software modules, manage models and patterns, and deploy them on cloud platforms. Special emphasis is placed on important frameworks like TensorFlow and PyTorch, alongside essential libraries such as Scikit-learn, Pandas, Matplotlib, and SciPy used for building learning architectures, data visualization, and model interpretation.
Program:
- Introduction to Learning Frameworks: TensorFlow, Keras, PyTorch
- Building Blocks of Learning: loss functions, activation functions, optimizers, vanishing gradients problems, batch normalization, regularization, dropout
- Custom Models: tensors and operations, tensors and NumPy, type conversions, variables, data structures, custom loss functions, custom metrics
- Loading and Preprocessing Data: shuffling, parsing, preprocessing features, chaining transformations
- Advanced Methodologies: meta-learning, transfer learning, domain adaptation, continual learning, active learning, multi-task learning, federated learning, Large Language Models (LLMs), Visual Language Models (VLMs)

Bibliography

Didactic methods

Lectures, laboratory experiences, exercises.

Learning assessment procedures

------------------------
UL: Reinforcement Learning
------------------------
To pass the exam, students will have to demonstrate that they:
- understand the principles behind how reinforcement learning and methods for programming modules based on artificial intelligence work
- be able to expose concepts of reinforcement learning and programming of modules based on artificial intelligence in a precise and organic way without digressions,
- knowing how to apply the acquired knowledge to solve application problems presented in the form of exercises, questions and projects.

The exam consists of a written test on the topics covered in the course. In case of low participation, the written exam will be replaced by an oral exam with equivalent questions. The questions may concern both the theoretical part and the exercises carried out in the laboratory. The written test can be followed by the development of a project.

------------------------
UL: Advanced programming for AI
------------------------
To pass the exam, students must demonstrate a comprehensive understanding of the principles underlying methods for programming modules based on artificial intelligence. They need to articulate the programming of AI modules clearly and cohesively, avoiding digressions. Furthermore, they must apply their acquired knowledge to solve application problems presented through exercises, questions, and projects.

The exam includes an oral assessment and evaluation of their solutions to lab exercises. The oral exam encompasses both theoretical concepts and practical exercises conducted in the lab. Additionally, with prior agreement from the teacher, students will choose to undertake a relevant project and submit a project report detailing their results. The project will be presented at the same time with the oral exam.

Evaluation criteria

Theoretical and applied knowledge of the techniques taught in the course; critical ability to select techniques based on the problem; ability to use the techniques taught in the course.

Criteria for the composition of the final grade

The final grade is represented by the arithmetic average of the grades of the two parts (RL / Advanced programming for AI) of the course.

Exam language

Inglese

Sustainable Development Goals - SDGs

This initiative contributes to the achievement of the Sustainable Development Goals of the UN Agenda 2030. More information on sustainability