A guide to Flutter Semantics

Any modern mobile application built for public use needs to support a large and diverse userbase. Mostly we think of designing apps for people of various ages, backgrounds, using a variety of hardware supporting a range of technical features. But that also includes people with various impairments which make usage of small displays cumbersome. According to some statistics these are not miniscule populations. For example, the number of visually impaired internet users in the US alone is larger than the whole userbase of Canada. Many of these people turn to accessibility features of their devices for help. Both Android and iOS provide a number of such capabilities – including font size and contrast manipulation, screen magnification, animation overrides, and others.

One of the popular accessibility features is screen reading, which allows users to navigate through visual elements by a number of gestures, and provides audible descriptions of application widgets. I urge everybody who wonders how these work by enabling TalkBack or VoiceOver on their mobiles and step into the shoes of many millions of people that rely on these tools every day. You’ll discover that your app has a secret facet that you haven’t even thought about.

You might be wondering if your Flutter app is going to make use of screen reader capabilities. The answer is (obviously): yes. There are several methods that one can use to help users in enhancing their experience. But before I describe how to do that, you need to check if your app doesn’t already behave in the way you want. Many of the widgets provided by Flutter have accessibility features implemented, and only if you are not satisfied with the default results should you try manually wrangling your app’s accessibility.

Flutter’s Semantics widget

The Flutter SDK doesn’t provide a fine-grained widget for controlling just accessibility. Instead, it contains a class called Semantics, with its description:

A widget that annotates the widget tree with a description of the meaning of the widgets.

As you can see, the widget doesn’t really care about the specific technology or feature, but rather describes a general information about the widgets it wraps. That said, most of the features are tailored for the two screen readers mentioned in the previous section.

When you check its constructor, you can get a sense of how many potential attributes it can take (and features it can support). Most of them end up in the SemanticsProperties class. I won’t describe them all here – but let’s take a look at several common ones.

Label, value & hint

One of the common use cases for the Semantics widget is to add a description to your widgets. In Semantics, we have (among others) three useful String parameters for that:

  • label, which is a brief description of the widget,
  • hint, which is an explanation of the action that will occur when the widget is interacted with,
  • and value, which provides a textual description of the widget’s state.

Naturally, you might not need all of them, but in some cases all of them fit. Let’s see one case where they do:

For the following text (which in fact is also a button, since we specify onTap handler), a user of screen reader will – assuming that the _counter variable stores zero, hear Counter button, zero, press to increase.

onCut, onCopy & onPaste

A useful bunch of properties that can help when interacting with text fields: onCut, onCopy, and onPaste are callbacks that get executed when user executed matching actions on text fields. You can use them to enhance user experience by providing a visual or audible feedback when user manipulates text.

The above code doesn’t do much, especially from the end user’s perspective, but you should get the idea.

Combining a11y features

While Semantics allows us to provide many options for adding accessibility features, we also need tools to suppress, or modify the already existing ones. Luckily, Flutter provides several mechanisms for that.

ExcludeSemantics

Let’s revisit the first example. I lied a bit when I wrote that it’s going to be read as:

Counter button, zero, press to increase.

In fact, you’re going to hear:

Counter button, zero, zero, press to increase.

Where did the second zero come from? As I mentioned, many Flutter widgets come with semantics already embedded in them, and so does Text. By default Flutter will attempt to combine semantic information from multiple widgets into one. To prevent this from happening, you can use the ExcludeSemantics class, which makes Flutter discard all the semantic information from the widget subtree starting with ExcludeSemantics’ child. See this example below on how to get rid of the extra zero.

There are other cases where you might want to use this widget. Say, you don’t like the built-in features of widgets provided by Flutter – while useful they might not fit all cases, and sometimes be very verbose.

BlockSemantics

While ExcludeSemantics works in many cases, you can imagine how cumbersome would it be, to add it whenever you open a popup in order to hide the background widget’s features. In such cases, it’s better to use BlockSemantics, which is a more convenient way to exclude semantics of sibling widgets. It’s smart enough to only do that for widgets that were painted before the BlockSemantics child, which means that you don’t need to manually determine which ones should, and which shouldn’t be ignored.

Let’s take a look at an example:

This Stack composes three rectangular containers with slight overlap between the siblings.

If you have a screen reader enabled, all three containers will be selectable. What would happen if we wrap each of the Semantics widgets in BlockSemantics?

  • Wrapping the “red” widget will not change a thing, as it’s painted first.
  • Wrapping the “green” widget will prevent the “red” widget from being selectable. However, the “blue” one, as it’s painted after the “green” widget, is still going to be selectable.
  • Wrapping the “blue” widget will make disable semantics on both other widgets. Notice, that this also includes the “red” widget, with which the “blue” one doesn’t overlap. This is because what matters in the painting order, not the area taken by the rendered widgets.

MergeSemantics

The last of the useful Semantics widgets is MergeSemantics. Imagine that you have a container which appears as a single widget, but actually is composed of multiple widgets. You might want to avoid having multiple semantically-annotated elements, but not by dropping some information, but rather combining them into one.

Let’s take a look at our previous example. If we wrapped the whole Stack in MergeSemantics, it would appear to the screen reader as a single item with combined description Red, Green, Blue.

Under the hood

If you’re curious, you might wonder how these widgets actually work. Flutter, apart from a tree of Widgets, keeps a separate structure called the semantics tree, which gets updated after the widgets are built, laid out, and painted.

The Semantics tree is composed from SemanticsNode objects, which are generated from RenderObject’s properties and SemanticsConfiguration (which, in turn, closely resembles SemanticProperties mentioned above). Not every RenderObject has its own SemanticsNode, rather they are maintained at certain points in the tree, called the semantics boundaries. RenderObjects that are not themselves a semantic boundary, have their configuration (if any) merged into the nearest parent boundary, or discarded (for example, if you use ExcludeSemantics).

showSemanticsDebugger

Whew, that’s a lot of info. One thing we haven’t covered yet is: what to do when something we can’t get something right? The Flutter SDK has a simple tool to let us examine the semantics tree as we progress through the app. To enable it, you need to add the showSemanticsDebugger: true argument to the app’s top WidgetsApp class (or its subclasses). Below you can see how this looks like for the default Flutter app.

On the left-hand side, you can see the regular view, on the opposite, the view with semantics debugger enabled

Summary

We went through some of the ways we can control accessibility features in Flutter using the *Semantics widgets. The topic of accessibility is, however, much larger than what fits into a single blog post. Sadly, Google doesn’t provide a concise and complete guide on that topic, that you can use. Instead, I recommend that you also take a look at these articles to get some more knowledge on the topic:

And remember – if you can’t figure something out, join the community and ask away :)

I’d like to thank Michał Pierzchała for his input on this article.

Should your next App be written in Flutter?

This post appeared originally on medium.com. Hence, the comments here are disabled – head there for discussion.

If you are a mobile developer, you must have noticed a bit of fuss around Google’s new, shiny SDK: Flutter. And if you are a mobile developer interested in creating a new app, you might have wondered: is it a viable option for your next project? In nearly every event that I’ve been to that touches on Flutter, I heard similar questions: Is Flutter mature enough for production? Does it support my favorite API? Why would I choose it over my current SDK? Let’s examine these concerns a bit closer.

New, or too new?

Unless you are just experimenting with different tech stacks, you probably think of releasing your new app to the wide world at some point. And you really don’t want to get stuck at the most tedious step of the development lifecycle, discovering that pushing your bundle to customers requires black magic.

Well, don’t fret! While Flutter’s official page lists just several major showcase apps, there’s a community-driven list of apps already in the wild that use the SDK. Surely people releasing those are not sorcerers! It’s doable and it’s being done.

But if you can’t stop wondering how the deployment of your Flutter app looks like, you need to remember that essentially you’ll be releasing a regular Android and/or iOS app with your Flutter code compiled into native libraries. Google has already provided some guides on how to do that on for iOS and Android, and several CI services (such as Bitrise, and Cirrus) have explicit support for Flutter apps. There’s even one service dedicated exclusively to Flutter.

More than just shiny

When Google presents Flutter, they like to show off insane animations, and repeat its impressive 60 fps rendering as a key feature of their SDK. But we in the real world know that eye-candy is often just an addition to the core functionality of our products. What good is 60 fps if you can’t integrate with your backend? You might as well sell fancy bricks.

Fortunately, Google maintains a listing of Flutter-compatible packages, that you can search through and explore. Many of those are written by Google engineers if you are wondering what’s the quality of those (if you don’t believe, check out their repository pages). And you can see there’s a lot of various packages — ranging from state management through UI features to operations support.

Naturally, that doesn’t mean that everything is already provided. If you depend on an obscure data format, you might need to write a bridge between the native code and your Flutter code using platform channels (shameless plug — see also my nifty presentation on that). And if you want to reuse your fancy UI, rather than rewriting it in Flutter, you can do that as well!

No painkillers necessary

With all this focus on what you can do, it’s easy to forget about You. While the features and possibilities of Flutter can be encouraging, you need to know if developing with it is going to hurt.

I need to be very frank here — the Flutter toolset is one of the main reasons I fell in love with it. First of all, the flutter command line tool is so stress-free that I’m eager to start new projects with it. Frankly, you might not even need more than it, but if you’re more inclined towards having everything done in your IDE, you can do that as well with plugins for Android Studio/IntelliJ and VS Code.

The development process itself is also nifty! With hot reload you can instantly see the changes to the widgets while you code. It preserves your widgets’ state, so there’s no need for redoing any operations that led you to your view. Of course, this comes with a downside: if you need the state to be reset, you’ll need to restart your app.

Moreover, Flutter comes with a set of tools to assist you in debugging — especially when it comes to the visual aspects of your app.

Do as Shia LaBeouf says

All of these arguments might be convincing, but the ultimate argument lies in practice. So find yourself a fun task, and try Flutter out! I’m pretty sure that after several hours of working with it, you’ll feel that it’s been the right thing to do.

As with any software, Flutter SDK is not flawless. Remember to reach out to the growing community if you feel stuck. Have fun, and see you there!