Django: The perils of string_if_invalid
in templates
Django’s template engine has a string_if_invalid
option that replaces missing variable lookups with a string of your choice:
TEMPLATES = [
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
# ...
"OPTIONS": {
# ...
"string_if_invalid": "MISSING VARIABLE %s",
},
}
]
The %s
will be replaced with the name of the missing variable.
This exists as a debugging aid to track down missing variables, but the documentation comes with a hefty warning:
For debug purposes only!
While
string_if_invalid
can be a useful debugging tool, it is a bad idea to turn it on as a 'development default'.Many templates, including some of Django's, rely upon the silence of the template system when a nonexistent variable is encountered. If you assign a value other than
''
tostring_if_invalid
, you will experience rendering problems with these templates and sites.Generally,
string_if_invalid
should only be enabled in order to debug a specific template problem, then cleared once debugging is complete.
(This warning was added in 2006 by Russell Keith-Magee in commit 73a6eb8.)
Despite the admonition, there are some recommendations out there to enable the option permanently in tests, including pytest-django’s --fail-on-template-vars
option and from myself in a post last year.
I’ve recently been exploring ways to prevent missing template variables for my client Silvr. As part of this, I used string_if_invalid
in more depth and came to learn just how much it can break template rendering. I changed my mind and would now follow the admonition in tests.
I wrote a script collecting ten examples of different ways that string_if_invalid
breaks template rendering. This post will walk through those examples.
If you want to run the script yourself, here’s the source:string_if_invalid_examples.py
sourcefrom contextlib import contextmanager
import django
from django.conf import settings
from django.template import engines
from django.template import Context
from django.template import Library
from django.template import Template
from django.urls import path
register = Library()
settings.configure(
TEMPLATES=[
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
"OPTIONS": {"builtins": [__name__]},
}
],
ROOT_URLCONF=__name__,
)
django.setup()
engine = engines["django"].engine
def page(request, name):
pass
urlpatterns = [path("<str:name>/", page, name="page")]
examples = [
(
"The default filter is overridden.",
"{{ is_tall|default:True }}",
{},
),
(
"The yesno filter is overridden.",
"{{ is_grande|yesno:'grande,tall' }}",
{},
),
(
(
"{% filter %} applies to missed variables, so searches for "
+ "'MISSING VARIABLE' can fail to match."
),
"{% filter lower %}{{ x }}{% endfilter %}",
{},
),
(
(
"{% url %} encodes string_if_invalid, forming invalid URLs, and "
"making searches fail to match."
),
"{% url 'page' name|default:'home' %}",
{},
),
(
(
"{% url %} encodes string_if_invalid, forming a URL where an "
+ "exception would be raised."
),
"{% url 'page' name %}",
{},
),
(
"{% static %} encodes string_if_invalid, making searches fail to match.",
"{% load static %}{% static x %}",
{},
),
]
def data_alterer():
pass
data_alterer.alters_data = True
examples.append(
(
"For functions marked alters_data=True, %s is not filled in.",
"{{ data_alterer }}",
{"data_alterer": data_alterer},
)
)
def needs_arg(x):
pass
examples.append(
(
"For function calls missing arguments, %s is also not filled in.",
"{{ needs_arg }}",
{"needs_arg": needs_arg},
)
)
class Silent(Exception):
silent_variable_failure = True
def raise_silent():
raise Silent()
examples.append(
(
"For silent variable exceptions, %s is also not filled in.",
"{{ raise_silent }}",
{"raise_silent": raise_silent},
)
)
@register.simple_tag
def yes_or_no(x):
if x:
return "Yes"
else:
return "No"
examples.append(
(
(
"Custom simple tags receive the value of string_if_invalid, which"
+ " can change their behaviour."
),
"{% yes_or_no x %}",
{},
)
)
@contextmanager
def patch_string_if_invalid(value):
engine.string_if_invalid = value
try:
yield
finally:
engine.string_if_invalid = ""
if __name__ == "__main__":
for index, (comment, template, context) in enumerate(examples, start=1):
print(f"Example {index}")
print(f" {comment}")
print(f" template = {template!r}")
def render():
try:
return Template(template).render(Context(context))
except Exception as exc:
return exc
print(f" string_if_invalid = '', result = {render()!r}")
with patch_string_if_invalid("MISSING VARIABLE %s"):
print(
f" string_if_invalid = 'MISSING VARIABLE %s', result = {render()!r}"
)
print("")
Run in an environment with Django installed:
$ python string_if_invalid_examples.py
Alright, on with the examples.
Breakage examples
1. The default
filter
Template:
{{ is_tall|default:True }}
Result without string_if_invalid
:
True
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
MISSING VARIABLE is_tall
The default
filter is often used to select default values for missing variables. Enabling string_if_invalid
breaks this pattern because the template engine then skips filters.
2. The yesno
filter
Template:
{{ is_grande|yesno:'grande,tall' }}
Result without string_if_invalid
:
tall
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
MISSING VARIABLE is_grande
The yesno
filter shows one of two strings based on a variable’s boolean value. In normal conditions, you can rely on missing variables evaluating as False
and being replaced with the second string. But with string_if_invalid
enabled, the filter is skipped, as per example #1.
3. {% filter %}
around missing variables
Template:
{% filter lower %}{{ x }}{% endfilter %}
Result without string_if_invalid
:
(empty)
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
missing variable x
The filter
tag applies a filter to a block’s contents. When wrapping content that includes string_if_invalid
, it will also be filtered. This could prevent you from finding missing variables in the output, such as if using a case-sensitive search in the above example.
4. {% url %}
encoding with default
Template:
{% url 'page' name|default:'home' %}
Result without string_if_invalid
:
/home/
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
/MISSING%20VARIABLE%20name/
The url
tag constructs a URL, potentially with parameters. If one of those parameters is a missing variable, it will be replaced with string_if_invalid
which is then URL-encoded. Again, this could prevent you from finding missing variables in the output with an automatic search. This one is particularly tricky since most {% url %}
usage is within the href
attribute, so it won’t be visibly rendered, but links will be broken.
5. {% url %}
that would raise an exception
Template:
{% url 'page' name %}
Error without string_if_invalid
:
NoReverseMatch("Reverse for 'page' with arguments '('',)' not found. 1 pattern(s) tried: ['(?P<name>[^/]+)/\\\\Z']")
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
/MISSING%20VARIABLE%20name/
Since missing variables normally resolve to ''
, this fails to match the URL pattern, which requires at least one character. Enabling string_if_invalid
makes the variable resolve to a non-empty string, allowing the URL to be resolved to a nonsense value, as above. “Muting” an exception like this is pretty undesirable for a “debugging tool“.
6. {% static %}
encoding
Template:
{% load static %}{% static x %}
Result without string_if_invalid
:
(empty)
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
MISSING%20VARIABLE%20x
Like the first {% url %}
example above, the static
tag ends up URL-encoding the result from string_if_invalid
. If you were searching for your string_if_invalid
value in the output, you wouldn’t find it.
7. Functions marked with alters_data=True
un-named
Template:
{{ data_alterer }}
Setup code:
def data_alterer():
pass
data_alterer.alters_data = True
Context:
{"data_alterer": data_alterer}
Result without string_if_invalid
:
(empty)
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
MISSING VARIABLE %s
By default, if a variable lookup resolves to a function, Django’s template engine calls it. Functions can be marked with alters_data = True
to prevent this behaviour, as documented under Variables and lookups. In such cases, the template engine falls back to string_if_invalid
, but does not template it with the name of the missing variable. This could make it hard to determine the issue.
8. Function missing arguments un-named
Template:
{{ data_alterer }}
Setup code:
def needs_arg(x):
pass
Context:
{"needs_arg": needs_arg}
Result without string_if_invalid
:
(empty)
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
MISSING VARIABLE %s
Following the above, a lookup function call fails due to missing arguments, the template engine will again fall back to string_if_invalid
without templating the variable name.
9. Silent exceptions don’t name the variable
Template:
{{ raise_silent }}
Setup code:
class Silent(Exception):
silent_variable_failure = True
def raise_silent():
raise Silent()
Context:
{"raise_silent": raise_silent}
Result without string_if_invalid
:
(empty)
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
MISSING VARIABLE %s
If resolving a variable raises an exception that has silent_variable_failure = True
set, the template engine will also fall back to string_if_invalid
. But again, this pathway misses the variable name, making it less useful for debugging.
10. Custom tags receive the string_if_invalid
value
Template:
{% yes_or_no x %}
Setup code:
@register.simple_tag
def yes_or_no(x):
if x:
return "Yes"
else:
return "No"
Result without string_if_invalid
:
No
Result with string_if_invalid = 'MISSING VARIABLE %s'
:
Yes
Template tags, including custom ones, receive the resolved variable values and then act on them. If string_if_invalid
is set, this changes the value received by the tag and can thus change its behaviour, in perhaps rather unexpected ways.
Conclusion
Well, those are the examples I found. I am pretty sure there are more ways string_if_invalid
can break templates—I only stopped searching at ten because it’s a nice round number.
After considering what I found, I will avoid setting string_if_invalid
. I’ll also be looking for it in any projects I join and recommending against it. I’d even be tempted to say we should remove it from Django.
I’m playing with monkey-patching in alternative undefined variable behaviour, based on Jinja’s more flexible options. I have a prototype that can raise exceptions for missing variables, allowing missing variables to be found early. Perhaps this will turn into a proposal for Django.
Learn how to make your tests run quickly in my book Speed Up Your Django Tests.
One summary email a week, no spam, I pinky promise.
Tags: django