The teaching is organized as follows:

Learning objectives

The course aims to provide the fundamental concepts of systems for processing large amounts of data, together with the programming paradigms adopted by these systems, and the cloud tools used to create a processing pipeline. At the end of the course the student will have acquired the knowledge necessary to evaluate possible alternatives in designing the analysis of large amounts of data, considering the benefits and limitations of the different approaches. This knowledge will allow the student to: i) design solutions to analyze large amounts of data; ii) evaluate the resources needed for the designed solutions; iii) configure systems using local or cloud resources; iv) continue their studies independently in the development of advanced analyzes of large amounts of data.

Prerequisites and basic notions

Nobody

Program

Distributed systems theory
Fundamentals and taxonomy of distributed systems; inter-process communication models (RPC, gRPC); time and synchronization (logical and physical clocks); leader election algorithms and consensus protocols (Raft); data consistency models and the CAP Theorem.

Data center architecture and virtualization
Physical organization of a data center: racks, failure domains, and network topology. Evolution from bare-metal to hypervisor-based virtualization (Type 1 vs Type 2). Comparison between virtual machines and containers: kernel sharing, performance, and isolation trade-offs. Proxmox VE as a Type 1 hypervisor platform: KVM-based virtual machine provisioning, LXC container management, storage pools, and network bridge configuration.

Containerization with Docker
Docker internal architecture (Namespaces, Cgroups, Layering); management of images, networks, and persistent volumes; software lifecycle optimization strategies through multi-stage builds; declarative multi-container orchestration with Docker Compose.

Orchestration with Kubernetes
Cluster architecture (Control Plane and Worker Nodes); declarative resource management via configuration files (YAML); fundamental abstractions (Pods, Deployments, Services, ConfigMaps, Secrets); persistent storage (PersistentVolumes and PersistentVolumeClaims); Service Discovery, Load Balancing, and Ingress mechanisms. Role of Proxmox as the infrastructure layer for on-premise Kubernetes clusters: creation of VM-based worker nodes via kubeadm, CNI plugin installation, and cluster lifecycle management. Comparison between self-hosted Kubernetes on Proxmox and cloud-managed offerings (AKS, EKS, GKE): operational trade-offs, cost models, and vendor lock-in.

NoSQL and distributed databases
Taxonomy of non-relational databases: document, key-value, column-family, and graph models. Consistency trade-offs in distributed databases read through the CAP Theorem lens. Deployment and interaction with NoSQL systems on containerized infrastructure.

Distributed storage and the MapReduce paradigm
Architecture of distributed filesystems (HDFS): NameNode, DataNode, and replication strategies. The MapReduce programming model: map, shuffle, and reduce phases. Hadoop ecosystem overview. Object storage as a cloud-native alternative: the S3 protocol and its role in data lake architectures.

Big Data processing with Apache Spark
Motivation for in-memory processing over disk-based MapReduce. Spark architecture: Driver, Executors, and SparkContext. Resilient Distributed Datasets (RDD) versus DataFrames. Lazy evaluation, transformations, and actions. Spark SQL, aggregations, and distributed joins. Introduction to Spark MLlib for distributed machine learning pipelines. Deployment of Spark workloads on Kubernetes via spark-submit.

IoT systems and distributed protocols
IoT reference architecture: perception layer, network layer, and application layer. Edge, fog, and cloud computing: latency, bandwidth, and processing trade-offs across the computational continuum. Constrained device programming: MQTT broker model and QoS levels, CoAP for RESTful semantics on resource-limited hardware, and AMQP for enterprise messaging. Security challenges in IoT deployments: device authentication, end-to-end encryption, and fleet management at scale. End-to-end pipeline design from sensor to dashboard: containerized MQTT brokers, stream processing with Spark Structured Streaming, time-series storage with InfluxDB and Prometheus, and operational visibility via Grafana.

Observability and AI model serving
Metrics, logs, and traces as the three pillars of observability. Prometheus architecture (pull-based scraping) and PromQL for cluster metrics analysis. Grafana dashboards for operational visibility. Model serving patterns: containerizing scikit-learn and FastAPI models, exposing REST inference endpoints on Kubernetes, and managing model versioning.

Didactic methods

- Lectures: presentation of theoretical models of distributed systems, coordination logic, and container-based isolation techniques. Theoretical sessions utilize interactive digital tools to encourage immediate discussion in the classroom and assess the level of understanding of the concepts covered.
- Methodological exercises: sessions dedicated to solving synchronization problems, analyzing coherence models, and consensus algorithms.
- Computer lab: practical implementation of the designed models using a containerization environment.

Learning assessment procedures

Discussion of a project developed by the students and agreed upon with the instructor. Further details will be provided during the course and published on Moodle.

Evaluation criteria

Methodological rigor in design
Ability to translate an application requirement into a consistent distributed model isolated via containers, demonstrating mastery of the derivation rules for microservices systems.

Mastery of languages and tools
Syntactic and logical correctness in the use of gRPC for communication, Docker for managing persistence and software packaging, Kubernetes manifests (YAML) for orchestration, and Proxmox for infrastructure virtualization.

Big Data systems design
Ability to select the appropriate distributed storage and processing model (HDFS, NoSQL, Spark) given an application requirement, justifying the architectural trade-offs.

IoT systems design
Ability to design an end-to-end IoT ingestion and processing pipeline, selecting appropriate protocols (MQTT, CoAP) and deployment strategies across the edge-fog-cloud continuum.

Analysis and automation skills
Ability to structure complex operational flows through code, with particular attention to observability, persistence management, and problem-solving in cloud-native contexts.

Technical terminology
Correct use of specific technical terminology in the field of intelligent systems engineering and distributed information systems.

Problem solving
Ability to apply theoretical concepts (such as consensus, load balancing, or data partitioning) to practical case studies derived from professional contexts.

Criteria for the composition of the final grade

The test is worth a maximum of 30 points.
Honors will be awarded to those who achieve a final score of 30 or higher.

Exam language

Inglese

Sustainable Development Goals - SDGs

This initiative contributes to the achievement of the Sustainable Development Goals of the UN Agenda 2030. More information on sustainability