November 09, 2020

ML misconceptions (9): neural networks are not black boxes

by Sam Sandqvist

The are a lot of misconceptions regarding neural networks. This article, and subsequent ones on the topic will present the major ones one need to take into account.(1)

Blog 16-3
By itself a neural network is something of a black box. Fully trained it will accept inputs, and produce outputs that hopefully makes sense. But nobody knows how it arrives at that particular outcome, or how it reasons. This is a problem.

This presents problems for people wanting to use them. For example, fund managers wouldn't know how a neural network makes trading decisions, so it is impossible to assess the risks of the trading strategies learned by the neural network.

Similarly, banks using neural networks for credit risk modelling would not be able to justify why a customer has a particular credit rating, which is a regulatory requirement.

That having been said, state of the art rule-extraction algorithms have been developed to make NNs more transparent, at least some neural network architectures. These algorithms extract knowledge from the neural networks as either mathematical expressions, symbolic logic, fuzzy logic, or decision trees.

This is shown in the figure below.

Blog 16-2

There are a number of possibilities to make a neural network transparent. Let’s have a look at some of them.

Mathematical rules

Algorithms have been developed which can extract multiple linear regression lines from neural networks.

The problem with these techniques is that the rules are often still difficult to understand, therefore these do not totally solve the 'black-box' problem - even if they fulfil the regulatory requirement.

There is no requirement for the rules to be simple or understandable! But of course if would be preferred if they were.

Propositional logic

Propositional logic is a branch of mathematical logic which deals with operations done on discrete valued variables. These variables, such as A or B, are often either TRUE or FALSE, but they could occupy values within a discrete range e.g. {BUY,HOLD,SELL}.

Logical operations can then be applied to those variables such as OR, AND, and XOR. The results are called predicates which can also be quantified over sets using the exists (∃) or for-all (∀) quantifiers. This is the difference between predicate and propositional logic.

If we had a simple neural network which Price (P), Simple Moving Average (SMA), and Exponential Moving Average (EMA) as inputs and we extracted a trend following strategy from the neural network in propositional logic, we might get rules like this…


Of course, we would need a very large number of these conditions in order to capture the whole network.

Fuzzy logic

Fuzzy logic is where probability and propositional logic meet.

The problem with propositional logic is that is deals in absolutes, e.g., BUY or SELL, TRUE or FALSE, 0 or 1. Therefore for e.g. traders there is no way to determine the confidence of these results.

Fuzzy logic overcomes this limitation by introducing a membership function which specifies how much a variable belongs to a particular domain. For example, a company (GOOG) might belong 0.7 to the domain {BUY} and 0.3 to the domain {SELL}. Combinations of neural networks and fuzzy logic are called neuro-fuzzy systems.

Decision trees

Decision trees show how decisions are made when given certain information.

We can evolve security analysis decision trees using genetic programming. Decision tree induction is the term given to the process of extracting decision trees from neural networks.

An example of a simple trading strategy represented using a decision tree.

Blog 16-3

The triangular boxes represent decision nodes, these could be to BUY, HOLD, or SELL a company. Each box represents a tuple of <indicator, inequality,="" value="">. An example might be <sma,>, 25> or <ema, <="," 30=“">.

In this way any output could be traced using the tree to its determining factors. Again, these trees might be huge and not represent a great improvement over a simple list of the conditions, as in the propositional logic example above.

All in all, the question of explainability and transparency in neural network use is not a simple one. So, even if it is a misconception that neural networks are black boxes, they do look awfully grey.

(1) The inspiration for the misconceptions is adapted from an article by Stuart Reid from 8 May 2014 available at

Sam Sandqvist

Sam Sandqvist

Dr Sam Sandqvist is our in-house Artificial Intelligence Guru. He holds a Dr. Sc. in Artificial Intelligence and is a published author. He is specialized in AI Theory, AI Models and Simulations. He also has industry experience in FinServ, Sales and Marketing.