Symbolic Knowledge Extraction and Injection:
Theory and Methods

(last built on: 2025-07-02)

Giovanni Ciatto and Matteo Magnini
Dipartimento di Informatica — Scienza e Ingegneria (DISI), Sede di Cesena,
Alma Mater Studiorum—Università di Bologna

Mini-school @ WOA 2025, the 26th Workshop From Objects to Agents
2nd July 2025, Trento, Italy

Link to these slides

https://MatteoMagnini.github.io/talk-2025-woa-nesy/

printable version

Why the wording “Symbolic” vs. “Sub-symbolic”? (pt. 1)

Local vs. Distributed Representations

• Local $\approx$ “symbolic”: each symbol has a clear, distinct meaning

  • e.g. "bear" is a symbol denoting a crisp category (either the animal is a bear or not)

• Distributed $\approx$ “non-symbolic”: symbols do not have a clear meaning per se, but the whole representation does

  • e.g. "swim" is fuzzy capability: one animal may be (un)able to swim to some extent

Let’s say we need to represent $N$ classes, how many columns would the tables have?

Why the wording “Symbolic” vs. “Sub-symbolic”? (pt. 2)

What is a “symbol” after all? Aren’t numbers symbols too?

According to Tim van Gelder in 1990:

Symbolic representations of knowledge

• involve a set of symbols
• which can be combined (e.g., concatenated) in (possibly) infinitely many ways,
• following precise syntactical rules,
• where both elementary symbols and any admissible combination of them can be assigned with meaning

Why “Sub-symbolic” instead of “Non-symbolic” or just “Numerical”?

• There exist approaches where symbols are combined with numbers, e.g.:

  • Probabilistic logic programming: where logic statements are combined with probabilities
  • Fuzzy logic: where logic statements are combined with degrees of truth
  • Bayesian networks: a.k.a. graphical models, where nodes are symbols and edges are conditional dependencies with probabilities, e.g.

• These approaches are not purely symbolic, but they are not purely numeric either, so we call the overall category “sub-symbolic”

Examples of Symbolic AI (pt. 1)

• Logic programming: SLD resolution (e.g., Prolog)
• Knowledge representation: Semantic Web (e.g., OWL), Description Logics (e.g., ALC)
• Automated reasoning: Theorem proving, Model checking
• Planning: STRIPS, PDDL

Examples of Symbolic AI (pt. 2)

Logic programming with SLD resolution

Examples of Symbolic AI (pt. 3)

Ontology definition in OWL

Examples of Symbolic AI (pt. 4)

Model-checking (as opposed to testing)

Examples of Symbolic AI (pt. 5)

Planning in STRIPS

Available actions

• grab(X): grabs block X from the table
• put(X): puts block X on the table
• stack(X, Y): stacks block X on top of block Y
• unstack(X, Y): un-stacks block X from block Y

Examples of Sub-symbolic AI (pt. 1)

• Machine learning: supervised, unsupervised, and reinforcement learning

  • Supervised learning: fitting a discrete (classification) or a continuous function (regression) from examples
  • Unsupervised learning: clustering, dimensionality reduction
  • Reinforcement learning: learning a policy to maximize a reward signal, via simulation

• Probabilistic reasoning: Bayesian networks, Markov models, probabilistic logic programming

Examples of Sub-symbolic AI (pt. 2)

Supervised learning

Examples of Sub-symbolic AI (pt. 3)

Supervised learning – Classification vs. Regression (1/2)

Data separation vs. curve fitting:

Examples of Sub-symbolic AI (pt. 4)

Supervised learning – Classification vs. Regression (2/2)

Focus on the target feature:

Examples of Sub-symbolic AI (pt. 5)

Unsupervised learning – Clustering

Examples of Sub-symbolic AI (pt. 6)

Unsupervised learning – Reinforcement learning (metaphor)

Examples of Sub-symbolic AI (pt. 7)

Reinforcement learning – Reinforcement learning (policy)

Supervised Machine Learning 101

Supervised Machine Learning 101 (pt. 1)

1. Let’s say you have a dataset of vectors in $\mathbb{R}^n$

2. Let’s say the vectors are labelled differently (2 colors)

3. Let’s say you want to separate them

2. Let’s say the vectors are aligned (along a curve)

3. Let’s say you want to interpolate them

Supervised Machine Learning 101 (pt. 2)

4. Then, need to approximate the function $f$…

4. … separating them (the sigmoid here)

5. The function $f$ is the decision boundary (a.k.a. classifier)

4. …interpolating them (the parabola here)

5. The function $f$ is the regression curve (a.k.a. regressor)

6. How to compute such a function $f$?

Supervised Machine Learning 101 (pt. 3)

Let’s formalize the problem (1/2)

1. Let’s assume the data represented as a set of vectors in a dataset $D \subset \mathbb{R}^n$ s.t. $|D| = m$
   • e.g. $D = {(x_1, y_1, c_1), (x_2, y_2, c_2), \ldots, (x_m, y_m, c_m)}$ for the classification dataset
   • e.g. $D = {(x_1, y_1), (x_2, y_2), \ldots, (x_m, y_m)}$ for the regression dataset
2. Let $\mathcal{X}$ (resp. $\mathcal{Y}$) be the input (resp. output or target) space of the data at hand
   • so $\exists X \in \mathcal{X}$ and $\exists Y \in \mathcal{Y}$ s.t. $(X, Y) \equiv D$
   • i.e. $X$ is the input data and $Y$ is the target data in $D$

Supervised Machine Learning 101 (pt. 4)

Let’s formalize the problem (2/2)

3. Let $\mathcal{H} = \lbrace f \mid f: \mathcal{X} \to \mathcal{Y}\rbrace$ be the set of all possible functions mapping $\mathcal{X}$ to $\mathcal{Y}$
   • i.e. the so-called hypothesis space
4. We need now to find the best $f^* \in \mathcal{H}$
   • best w.r.t to what?
5. We need to define a loss function $\mathcal{L}$ that quantifies the difference between the predicted output $f(X)$ and the true output $Y$
   • for any possible $f \in \mathcal{H}$
6. So, our learning problem is as simple as a search problem in the hypothesis space: $$ f^* = \arg\min_{f \in \mathcal{H}} \mathcal{L}(f(X), Y) $$
7. Ok, but in practice, how do we do that?

Supervised Machine Learning 101 (pt. 5)

Let’s solve the problem

1. How to explore the hypothesis space, depends on what sorts of functions it contains
2. There exist several sorts functions for which an exploration algorithm is known
   • e.g. $\mathcal{H}$ $\equiv$ polynomials of degree $1$ (i.e. lines: $f(\bar{x}) = \beta + \bar{\omega} \cdot \bar{x} = \beta + \omega_1 x_1 + \ldots + \omega_n x_n$)
   • e.g. $\mathcal{H}$ $\equiv$ polynomials of degree $2$ (i.e. parabolas: $f(\bar{x}) = \beta + \omega_1 x_1 + \ldots + \omega_n x_n + \omega_1^2 x_1^2 + \ldots + \omega_n^2 x_n^2$)
   • e.g. $\mathcal{H}$ $\equiv$ decision trees (i.e. if-then-else rules: $f(\bar{x}) = \text{if } x_1 \leq c_1 \text{ then } y_1 \text{ else if } x_2 \leq c_2 \text{ then } y_2 \ldots$)
   • e.g. $\mathcal{H}$ $\equiv$ $k$-nearest neighbors ($f(\bar{x}) = \text{class of (most of) the } k \text{ nearest neighbors of } \bar{x}$)
   • e.g. $\mathcal{H}$ $\equiv$ neural networks (i.e. NNs: $f(\bar{x}) = \sigma(\beta + \bar{\omega} \cdot \bar{x})$ where $\sigma$ is a non-linear activation function, e.g. sigmoid, ReLU, etc.)

Supervised Machine Learning 101 (pt. 6)

Let’s solve the problem with neural networks

Single neuron

(Feed-forward)
Neural network $\equiv$ cascade of layers

Many admissible architectures

Supervised Machine Learning 101 (pt. 7)

Let’s solve the problem with gradient descent

• NNs (and other ML algorithms) explore the hypothesis space by using a gradient descent algorithm

• In this case:

  • the hypothesis space $\mathcal{H}$ $\equiv$ “the set of all possible NNs with a given architecture”
    • e.g. 4 input neurons, 2 hidden layers with 3 neurons each, and 1 output neuron, all layers fully connected, using ReLU activation functions
  • the loss function $\mathcal{L}$ $\equiv$ “the difference between the predicted output and the true output”
    • e.g. $\mathcal{L}(f(X), Y) = \frac{1}{n} \sum_{i=1}^{n} (f(x_i) - y_i)^2$ (mean-squared error) for regression
    • e.g. $\mathcal{L}(f(X), Y) = -\sum_{i=1}^{n} y_i \log(f(x_i)) + (1 - y_i) \log(1 - f(x_i))$ (cross-entropy) for classification
  • the algorithm initializes the weights of the NN randomly, and then iteratively updates them by minimising the gradient of the loss function w.r.t. the weights
    • i.e. $w_{i+1} = w_i - \eta \nabla_w \mathcal{L}(f(X), Y)$ where $\eta$ is the learning rate
  
  • the approach is greedy and sensitive to local minima, but very effective in practice

Supervised Machine Learning 101 (pt. 8)

How things may go wrong – Underfitting vs. Overfitting

Supervised Machine Learning 101 (pt. 9)

How to address: Test-Set Separation

Supervised Machine Learning 101 (pt. 10)

How things may go wrong – Training is stochastic

Depending on its start, training may yield different results:

Solution is coss-validation:

1. Train the $k$ different models (same architecture) on $k$ different subsets of the training set
2. Average the results of the $k$ models
3. If average is good, the model is considered more robust and reliable
   • useful when comparing different architectures or hyper-parameters

Supervised Machine Learning 101 (pt. 11)

How things may go wrong – Data may not be numeric

Solution: change encoding

(Ordinal encoding)

(One-hot encoding)

Supervised Machine Learning 101 (pt. 12)

How things may go wrong – Data may not be separable by a proper function…

Solution: feature engineering (map data into a different space)

Non-linearly-separable data …
$(x, y)$

… may be separable in another space
$(\rho = \sqrt{x^2 + y^2}$, $\theta = \arctan(y/x))$

Definition and Motivation (pt. 1)

any algorithmic procedure accepting trained sub-symbolic predictors as input and producing symbolic knowledge as output, so that the extracted knowledge reflects the behaviour of the predictor with high fidelity.

Definition and Motivation (pt. 2)

• Explainable AI (XAI): SKE methods are often used to provide explanations for the decisions made by sub-symbolic predictors, making them more interpretable and understandable to humans (a.k.a. post-hoc explainability)

  • local explanations: explanations for individual predictions
  • global explanations: explanations for the overall behaviour of the predictor

• Knowledge discovery: SKE methods can help discover patterns and relationships in the data that may not be immediately apparent, thus providing insights into the underlying processes

• Model compression: SKE methods can simplify complex sub-symbolic models by extracting symbolic rules that approximate their behaviour, thus reducing the model’s size and complexity

Explainability vs Interpretability

They are not synonyms in spite of the fact that they are often used interchangeably!

Concepts

Main entities and how to extract symbolic knowledge from sub-symbolic predictors

Entities

Sub-symbolic predictor

Symbolic knowledge

Logic Rule
Class = setosa ← PetalWidth ≤ 1.0
Class = versicolor ← PetalLength > 4.9 ∧ SepalWidth ∈ [2.9, 3.2]
Class = versicolor ← PetalWidth > 1.6
Class = virginica ← SepalWidth ≤ 2.9
Class = virginica ← SepalLength ∈ [5.4, 6.3]
Class = virginica ← PetalWidth ∈ [1.0, 1.6]

How SKE works

CART (pt. 1)

Classification and regression trees (cf. Breiman et al., 1984)

CART (pt. 2)

An example decision tree estimating the probability of kyphosis after spinal surgery, given the age of the patient and the vertebra at which surgery was start ed (rf. wiki:dt-learning). Notice that all decision trees subtend a partition of the input space, and that those trees themselves provide intelligible representations of how predictions are attained.

CART (pt. 3)

1. generate a synthetic dataset by using the predictions of the sub-symbolic predictor

2. train a decision tree on the synthetic dataset

3. compute the fidelity and repeat step 2 until satisfied

4. [optional] rewrite the tree as a set of symbolic rules

Practical example of SKE

The Adult dataset (cf. Becker Barry and Kohavi Ronny, 1996)

Adult classification task

The Adult dataset contains the records (48,842) of individuals based on census data (this dataset is also known as Census Income). The dataset has many features (14) related to the individuals’ demographics, such as age, education, and occupation. The target feature is whether the individual earns more than $50,000 per year.

What we will do

1. Download the Adult dataset and preprocess it

2. Train a sub-symbolic predictor – a neural network – on the Adult dataset

3. Use a pedagogical SKE method – CART – to extract symbolic knowledge from the trained neural network

4. Visualise the extracted symbolic knowledge as a decision tree and as a set of rules

Jump to the code

GitHub repository at github.com/MatteoMagnini/demo-2025-woa-nesy/blob/master/notebook/extraction.ipynb

Taxonomy of SKE methods (pt. 1)

Taxonomy of SKE methods (pt. 2)

Taxonomy of SKE methods (pt. 3)

Discussion

Future research activities

• target images or highly dimensional data in general

• target reinforcement learning (when based on NN)

• target unsupervised learning

• design and prototype your own extraction algorithm

Definition and Motivation (pt. 1)

Any algorithmic procedure affecting how sub-symbolic predictors draw their inferences in such a way that predictions are either computed as a function of, or made consistent with, some given symbolic knowledge.

Definition and Motivation (pt. 2)

• Improve predictive performance: by injecting symbolic knowledge, we can

  • guide the learning process in order to penalise inconsistencies with the symbolic knowledge, or
  • structure the model’s architecture to mimic the symbolic knowledge

• Enhance interpretability: with SKI we can make predictors that are

  • interpretable by transparent box design, as they are built to mimic symbolic knowledge
  • interpretable using symbols as constraints, as they are built to respect symbolic knowledge

• Robustness to data degradation: symbolic knowledge can help sub-symbolic models maintain performance even in the presence of noisy or scarcity of data

• Enhance fairness: by incorporating symbolic knowledge about fairness constraints, we can ensure that sub-symbolic models make decisions that align with ethical considerations

• And more: SKI can simplify the predictor’s architecture, in particular it can reduce the number of weights in a neural network, thus improving its efficiency and reducing the risk of overfitting

Concepts

Main entities and how to inject symbolic knowledge into sub-symbolic predictors

Entities

• Predictor: a sub-symbolic model that makes predictions based on input data, usually a neural network

• Symbolic knowledge: structured, formal knowledge that can be represented in a symbolic form. The most common forms of symbolic knowledge are

  • Propositional logic, simple rules with if-then structure
  • Datalog, a subset of first-order logic with no function symbols, only constants and variables

• Fuzzification: the process of converting symbolic knowledge into a form that can be used by sub-symbolic predictors, e.g. by assigning degrees of truth to symbolic statements

• Injector: the main component that injects symbolic knowledge into the predictor, by modifying its architecture, its training process or by other means

Structuring

Constraining

Embedding

Knowledge Injection via Network Structuring (KINS)

(ref. Magnini et al., 2023)

Fuzzification

$^{*}$ encodes the value for the $i^{\text{th}}$ output $^{**}$ assuming $p$ is defined by $k$ clauses of the form: ${p}(\bar{X}) \leftarrow \psi_1,\ \ldots,\ {p}(\bar{X}) \leftarrow \psi_k$

Injector (pt.1)

Injector (pt. 2)

Knowledge Injection via Lambda Layer (KILL)

(ref. Magnini et al., 2022)

Fuzzification

$^{*}$ encodes the penalty for the $i^{\text{th}}$ neuron $^{**}$ assuming predicate $p$ is defined by $k$ clauses of the form: ${p}(\bar{X}) \leftarrow \psi_1,\ \ldots,\ {p}(\bar{X}) \leftarrow \psi_k$

Injector (pt.1)

Cost function: whenever the neural network wrongly predicts a class and violates the prior knowledge a cost proportional to the violation is added. In this way the output of the network differs more from the expected one and this affects the back propagation step.

Injector (pt. 2)

Practical example of SKI

The poker hand data set (PHDS) (cf. Cattral Robert and Oppacher Franz, 2002)

PHDS classification task

Logic rules to inject (pt. 1)

Logic rules to inject (pt. 2)

Logic rules to inject (pt. 3)

What we will do

1. Download the PHDS dataset and preprocess it

2. Define the symbolic knowledge to inject

3. Train a sub-symbolic predictor – a neural network – on the PHDS dataset

4. Train a second neural network with the symbolic knowledge injected

5. We will inject the knowledge in the loss function

6. Visualise and compare the results of the two predictors

Jump to the code!

GitHub repository at github.com/MatteoMagnini/demo-2025-woa-nesy/blob/master/notebook/injection.ipynb

Taxonomy of SKI methods (pt. 1)

Taxonomy of SKI methods (pt. 2)

• input knowledge: how is the knowledge to-be-injected represented?

  • commonly, some sub-set of first-order logic (FOL)

• target predictor: which predictors can knowledge be injected into?

  • mostly, neural networks

• strategy: how does injection actually work?

  • guided learning: the input knowledge is used to guide the training process
  • structuring: the internal composition of the predictor is (re-)structured to reflect the input knowledge
  • embedding: the input knowledge is converted into numeric array form

• purpose: why is knowledge injected in the first place?

  • knowledge manipulation: improve / extend / reason about symbol knowledge—subsymbolically
  • learning support: improve the sub-symbolic predictor (e.g. speed, size, etc.)

Discussion

Future research activities

• foundational
  address recursion

• practical
  find a language that is a good balance between expressiveness and ease of use

• target
  apply to large language models