But this is only a part of the problem, for although all this helps you to create answers to your scaling problems, it does not answer the problems that created the need for those answers: the fundamental part of the problem.

As we very well know, “Only when you know the question, will you know what the answer means”.  Luckily in this case the question is easy enough and it is: “How do I scale my application across many devices?” So we only have to answer it. Unfortunately, however, 42 does not work well enough in this case and the sad reality is that there is no single definitive answer.

Nonetheless, there are some “rules” or general concepts, that if taken into consideration when finding the answer, do improve the quality of it. These rules will be subject of this blogpost.

DESIGN once, evaluate every time.

So after we get to scale our elements in a meaningful correct way, as we discussed in the previous posts, one could think that it would be enough to just set some base position rules and be done with it. Truth is that changes in screen ratio when porting your application to a new device don’t always go hand in hand with resolution changes and require a repositioning of components rather than simple scaling of the said components.

This means that sometimes we have and should use a different ratio to reposition our assets. A clear example is portrait and landscape form factors. Portrait traditionally comes better with single column layout, where in landscape we can have multiple columns.

As we also talked about in earlier posts, the fundamental metric for objects in this space is the size of the user’s finger, so when a screen gets physically larger we get more space for more fingers, ergo we might want to use that in our app providing more explicit functionality in each screen.

Platform matters.

Porting your application to a specific platform does not occur in a vacuum. Most of the time each specific platform has extensive guidelines on how to tackle this, so even when we think we have a better answer, we have to understand that being right when everyone else is wrong is one of the worst experiences in this area. So try to comply with the user’s expectation in a specific platform.

It’s the principles that are meaningful not the pixels.

Your brand and experience concepts should stay regardless of screen/platform. This sentence is perpendicular to the previous point, and points out that as your application becomes available in multiple places you should not reinvent it at every turn. You should stay true with some predefined core image/interaction/experience principles that help define your brand and your application uniqueness across multiple points of interaction. You may need help – realise that this feels like and often resembles a perpetual exercise of “squaring the circle”. This space is evolving rapidly and the answers perpetually change, so seek help externally if you feel lost.

Start by not trying to make it perfect for all cases.

Android, iOS, Windows Phone, BB10, Tizen, Windows Desktops, Linux Desktops, OSX, embedded etc. – there are several hundreds of different screens out there. Group them into areas, don’t try to make it perfect for every case in every platform.
Start by discovering the target group of your application and choose a widely adopted resolution and platform as your start point. Define some simple base rules for scaling the application on the chosen platform, expand on the core principles developed there as needed. And then scale into other platforms.

Test, Test and then Test again.

Development and testing of your application on the Desktop ≠ the Device. This is vastly different in terms of experience and the final result. Test as much as possible – some things only make sense in the real thing.

The context is part of it.

Each platform is used in a different context and user mindset. A phone is used with the user on a specific position, location and attitude towards the iteration than for example a desktop. An embedded device can be used in extremely specific situations for example in an industrial plant. Use this info to evaluate how to scale each feature of your application on the canvas. Weight the content creation vs content consumption or the decision pattern in each case so that in each context the user might find what he is looking for. Users look for different things in different contexts.

In the end, Scaling is only one part.

Scaling your app correctly and efficiently is only part of the story. In fact the quality of the referred scaling is in great measure the result of a previously well-crafted User Experience, well defined User Scope, efficient Branding and good Usability.

This Concludes the series on Scalable Uis. I hope you found here information that might be useful for your work.
It is not by any means an exhaustive view on the subject nor a definitive answer. What I hope with this series is to make some of the issues a bit more present, so that you might avoid costly loops of unplanned revision and bitter surprises, and gain a bit more understanding about the several aspects of scaling multiple concepts.
I might revisit this topic if some new aspect surfaces that is interesting to talk about and relevant enough.

The post Scalable UIs, Scaling the Content appeared first on ACuteDesign.

To start this series of posts I had a main motivator, SVG.  It is a great file format, its the file type I use day in day out and the format I use the most to create all of my images…
But every so often the question about scalable UI’s and Vectors pops up. And someone will say something like “we should just use vectors to scale things”. To that, I will usually say something like, “Scalable Vectors Graphics are scalable but your screen is not“, hoping it will close the conversation just there, and it usually does.

However the above statement is only partly correct and is not the definitive reason why we should avoid off-the-shelf vectors as the source image format for our UI assets.

Scalable definition is a bit like being “impassioned”.

The way we define “Scalable” UI’s, as we have seen in the past posts, is very peculiar and we tend to use it the way it suits us best, ignoring the practical differences between the different meanings of the concept. Ergo, like being  impassioned, the target of our focus is more what we want it to be rather than what it really is.
This tends to produce the confusions and misunderstandings that are so common in this area, precisely because the Scalable part in SVG, is for a certain type of the referred concept, and most of the time not the type of scalable we need in UI.

So what does scalable mean for a Scalable Vector Graphic?

An SVG or any other main vector format is formed (mostly) of mathematical information about paths and its control points (Bézier curves), its visual size is only relevant in regards to the render size of the canvas it’s on, and as a result you can zoom an image almost infinitely and will never see pixels. (the pixels are just rendered for a given view area and change accordingly to the section of the vectors in that area).
This makes its a great format to scale images to really huge formats. A rendered 40000×40000 px image that is scaled down to 1000×1000 will look exactly like the image originally rendered in 1000×1000.
Now as we have seen so far, this is often not the type of scalable we want.

• Firstly, it is X/Y dependent where we want many times X/Y independently scalable elements (think of scaling a rounded rectangle), and for those we will need something like Borderimage components.
• Secondly, as you zoom out many elements become sub pixel and difficult to see though still there. For example, in maps we may want vectors that render differently depending on the zoom level, like making roads disappear/appear as you zoom out/in, as well as simplifying aspects of the path itself.
• Thirdly, it ignores pixels, as we mentioned pixels are still very important, in lower definition screens we can make use of the of the pixel to create sharp contrasts, this makes your visual element “POP Out” against the background. However, SVG, in its mostly perfect mathematical description of path positions, completely ignores the rendering grid and as a result can produce unsharp elements that are off the pixel grid by decimals of a pixel. (this is only problematic in rectangular elements that align with the pixel grid disposition, but we do use it as an advantage for designs).

SVG’s in QML.

You can use SVG in QML right now as a source format, but attention, it won’t be re-rendered unless you tell it to do that, the result will be that if you do a scale or even change width height you will end up seeing pixels.  You can create bigger renders that will provide higher definitions that are more zoomable, but at the cost of it taking more time to render it and taking a lot of memory for the cached image.
Also SVGs can be very complex, I have created SVGs that take several hours to render Many of my past wallpapers for KDE were done in outline mode and I would only occasionally look at them with filters and colors on and I do have a powerful desktop to render those; trying similar things on a mobile is not a great idea.
SVG support in QT is limited, many things won’t work, the filters are mostly not working so the look will dramatically change if you expect blur-based drop shadows to work, you will not see those, the same goes for multiply filters, opacity masks, etc, etc…
So, in a nutshell, don’t use SVG as a base source image unless you know its limitations and how it works, it’s a wonderful format if you understand it’s limitations and strengths, and is sometimes useful in QML.

^{Vector sunset wallpaper crop several hours to render on my old linux pc.}

What about other vector formats? like Fonts?

There is a special Vector format’s that we use all the time and that is also scalable, and its a format that has dealt with this problems many years ago, Fonts…

Fonts are special types of monochromatic vector paths that can have special hints to cater to pixel scalable issues. It does that via 2 methods

• Font hinting (also known as instructing) is the use of mathematical instructions to adjust the font’s visual appearance so that it lines up with a virtual grid.
• Font Kerning is moving the glyph on the relative x direction to achieve a more pleasing visual result and sometimes to help with the grid alignment issues…

All of this magic is done by your local font render engine and the extent to how much these operations are done depends on your local font rendering definitions…

Now, since the introduction of QML 2, the way fonts are rendered has changed, and, by default font hinting is ignored.  As a result, if you zoom into a font by animating the pixel size you get a nice smooth zoom effect, but slightly more blurry all around fonts, since they are not hinted.
QML does allow you to have native looking fonts by doing

Text {
    text: "Native here!"
    renderType: Text.NativeRendering
}

How ever if you try to do a zoom effect here, via animating the pixel size, you will have a “jumpy” text feeling because, as the size increases, the hinting instructions of the font will keep on trying to adjust to an ever-changing pixel grid.
Also, the default method does a vector like scaling via the distance field method when changing the  scale: property,  where when using native rendering, you see the pixels of the 1:1 scale ratio being scaled.
Another side effect of the distance field method used by default is that if the scale/font.size is very large you start to see the inaccuracies of the method, and this is valid for any font size you pick, the method generates the distance field glyph from a small image based on the font and it’s not updated as you scale it.

Also if the font is not well formatted (glyph bounding box smaller than the glyph itself) it might clip the font in some border areas with weird visual results.

So, my advice here is: if you need readable text that you don’t want to do pinch to zoom or any other effect that changes the font size in an animated way, use the native render. If you want something dynamic then go for the default mode. Or, you can even try for a compromise solution where, when animating, you choose default and in the end turn native on. It’s mostly a matter of what you consider most important and more polished at the end of the day.
Also noteworthy is that on the higher DPI screens the hinting instructions lose a lot of their importance since the pixel size and respective sub-pixel antialiasing ‘grays’ become much smaller and relatively less important in relation to the font body. The same is true for non square pixels like many _{(but not all)} AMOLED screens have.

Next!

Next post will return to the subject of making scalable X/Y independent elements that work well with DPI metrics…
By the way, we will be discussing this subjects at the training days of Qt Developer Days 2014 Berlin, if you are interested in this subjects Registration is here.

So see you soon, here on my next post or at DevDays.

The post Of vectors and scalable things. appeared first on ACuteDesign.

In the previous blog post, we talked about the importance of pixels and their indivisible nature, the way we deal with that in the icon area and introduced the concept of PPI dependable UIs – all this to have truly scalable graphics in QML.

We used the BorderImage element to illustrate how an element could scale and retain the same aspect ratio across a range of different screen PPIs and their corresponding screen definitions.

BorderImage is for Scaling!

When I first got into Qt Quick and QML, the one element that made me instantly fall in love with the entire thing was the BorderImage element. It made things that were complicated instantly trivial, plus it saved me, the designer, the tedious trouble of pre-slicing images into 9 rectangular elements, which dramatically helped with the “deliverables” problem and with small adjustments. Even today, with all of the new things we have like the Canvas or  Shaders, BorderImage remains my favourite element in QML.
However that might not be enough (and it really is not enough). To expose what goes on when ppi changes, I created a video on a hypothetical screen. Using similar code to that shown in the last post: a BorderImage component that lived inside an element whose pixel count grew, and which then, when scaled back into the original size, mimicked the scaled appearance of a flexible dpi count screen…
Here is the code that generates that central blue element:

.......
 BorderImage {
   id: scalableElement2
   source: "./Images/baseBImage.png"
   width: parent.width-ppi/2
   height: ppi*1.5
   anchors.centerIn: parent
   border.left: 27; border.top: 27
   border.right: 16; border.bottom: 16
 }
.......

         _{(notice it is obviously not pixel perfect as pixels are scaled)}

output31 _{(click here if your browser does not want to show the video)} output_file.webm

So as you can see the results are far from perfect. In spite of the created element retaining the desired overall size, its look radically changes as we grow the pixel count, the corner radius diminishes almost into nothing, and even the overly done drop shadow that is huge in 90dpi becomes almost invisible on an hypothetical 570 dpi screen.

An expected solution

One possible solution, and the most commonly used, gets its inspiration from the way we scale icons, this means using multiple source Image files for ppi ranges.

And the way we generate those source images is again very similar to the way we generate them for icons. Using a vector drawing program, we scale the source image into multiple scale size elements and render those into raster images, png’s, jpeg’s, etc.

We can also define the sizes of the corners/borders in ppi/inches.  As long as the ppi is larger than the number of pixels the corner has, we will be just extending the place where border line cuts the image. You just need to pay attention to leaving enough margin in those areas, so that no gradient/texture scaling problems might become too visible. Just keep in mind that the slicing lines of the border image will not be in a precisely defined position.

Showing the Code

What I will show now is just demo code. There are better ways to do it with components now offered in Qt to facilitate this, but here I am just trying to expose the logic and the problems, so I have opted for this simple method.

.......
property var dir: ["MDPI","HDPI","XHDPI","XXHDPI",
                                    "XXXHDPI","XXXXHDPI"]
readonly property int ppiRange:{ 
 if (ppi>=540)
  5
 else if (ppi>=360)
  4
 else if (ppi>=270)
  3
 else if (ppi>=180)
  2
 else if (ppi>=135)
  1
 else
  0
 }
.......

...... 
BorderImage {
   id: scalableElement3
   source: "./Images/" + dir[ppiRange] + "/image.png"
   width: parent.width-ppi/2
   height: ppi*1.5
   anchors.centerIn: parent
   border.left: 0.3*ppi; border.top: 0.3*ppi
   border.right: 0.18*ppi; border.bottom: 0.18*ppi
 }
.......

What you can see here is a simple way to change the source directory, depending on the dpi and the the Border position cuts defined in inches via the ppi number… I generate all the images and place them in conveniently named directories that are mapped in our “dir” variant. This has proven to be a generally good idea in my experience when creating all of the assets; it makes it easy to compare directories and search for missing assets.

Again, here is a video showing how it all looks when we dynamically change the ppi. Remember once more that the real thing will look much better since it will not be scaling pixels – but it should give you an idea of how it works.

output32 _{(click here if your browser does not want to show the video)} output32.webm

As you can see the results are much better now and the image retains its global appearance across the ppi range. There are still issues in the corners and shadow size, especially when reaching the 540 ppi area, and if you target a device that is only slightly below 540, you might want to create more assets for that segment. This is true for any case, the values I picked were somewhat arbitrary and a multiple of a 90 ppi theoretical screen. In real life you tend to target a specific set of devices and probably will want to use those as the ones that set your boundaries for the definition of your image assets. Just pay attention to where each range ends and another begins, and make sure you are not discarding important visual information.

In the next post we will discuss more exotic methods of scaling that might be useful, depending on what you want to scale.  So, see you soon.

The post BorderImage is for Scaling! Scalable UIs 2.2. appeared first on ACuteDesign.

In the earlier post, we talked about the difficulty of scaling Uis across the current PPI range we have on the mobile and embedded space. We also hinted that the most common way to avoid this problem was done via providing several image assets that help to mitigate this issues.

How does it work? How can one carry out it in QML? And more importantly where does it fail? Were it’s the best solution and what trade offs does it involve?

Icons.

Icons are probably the easiest place to start here, long before the mobile became what it is, we were scaling icons, very much in the same way we are doing those very same icons for mobile today.

Icons depending on the platform were done in 2x multiples of 16 and a size in between. So 16×16, 24×24, 32×32, 48×48, 64×64, 96×96, 128×128,172×172, 256×256, 512×152… The reasons for all these sizes are multiple and range from marketing, specific use, type, clickable area and visibility. But you might ask how come so many and how come we don’t just use one size and scale it?

To answer that question we first need to understand the Pixel and how scaling works.

What is a Pixel?

A pixel is a color composed of smaller colors, it’s the almost indivisible element, its shape depends on the technology used by the screen, but for simplicity’s sake we use the “mostly” square element made by the RGB strips on an LCD screen. We use this pixel because 1) it maps really well to image pixels, and 2) it’s the most common technology used in displays.

How does scale work?

Scaling can be done via multiple smart or not so smart algorithms but usually is the result of some interpolation of pixels in a new rendering grid. The result is that when a pixel is stretched it is distributed across the new pixel grid.

Something about icons.

Icons are special images that use the pixel grid to their full potential. In icons we still paint in pixels even if they are done in a vector drawing application. We still align our vectors to a virtual rendering grid, we do this to maximize contrasts and sharpness, and as a result icons tend to make a lot of use of perpendicular and horizontal contrasts.

Scaling an icon results in blurry images, as those pixel contrasts get blurred on the interpolating process and only work on multiple values if the contrast lines were done on the major 16×16 grid lines. Even there we lose meaningful data as a contrast that was 1 pixel wide becomes sub-pixel 0.5 or 0.25.

For this reason, when we do iconography, each icon is redone for every size, and we NEVER scale icons from one size to another…

As you can see in the image, scaling an icon by just 1 pixel can result in very blurry results. My advice is that if you need icons in different sizes that look the same in different pixels per inch, you should provide multiple images using 2x and one in between scale factors with the icon redone to take advantage of the extra pixels.

What Android expects for icons

Again using android as our example platform

“to create an icon for different densities, you should follow the 2:3:4:6:8 scaling ratio between the five primary densities (medium, high, x-high, xx-high, and xxx-high respectively). For example, consider that the size for a launcher icon is specified to be 48×48 dp. This means the baseline (MDPI) asset is 48×48 px, and the high density (HDPI) asset should be 1.5x the baseline at 72×72 px, and the x-high density (XHDPI) asset should be 2x the baseline at 96×96 px, and so on.”

 

_{notice that each icon was slight altered to take advantage of the extra pixels while maintaining the same look overall.}

Buttons and other X/Y independent scalable elements

What about UI elements that are supposed to scale in an X/Y independent fashion, like Buttons, GroupBoxes and other custom image-based elements. How do they work?

In QML we have a Component that was made with these sort of elements in mind, the BorderIMage, it expect an image that it can slice in 9 bits so it can scale/tile each one in an appropriate way. example code..

BorderImage {
   id: scalableElement
   source: "./image/Imagebase.png"
   width: 100; height: 100
   border.left: 25; border.top: 25
   border.right: 25; border.bottom: 25
  }

The code above creates an element with 100 by 100 pixels using unscaled corners with 25×25 pixels from Imagebase.png wile stretching the areas in between those 25 pixel segments.

Now for the main question? How big should it be? And in what metric?

If it’s used on a touch screen then it should be big enough for you to touch it without triggering false hits or miss hits, and big enough is something between 7 and 9 mm depending also on how tolerable a missed hit can be for the user. This number is roughly speaking the size of your thumb contact surface with a touch screen.

This means that we have two conflicting metrics here, on one end we have the pixel, on the other physical size, we still express values to the QML code in pixels and yet we need to make this elements as big as my thumb.

PPI or DPI

PPI or DPI are the amount of pixels in an inch of your screen, you can expose the PPI to QML via several methods, but we want the real PPI not the logical one. If you are “pixel crazy” like me, you will want the PPIx and PPIy – yes the pixels are not always really “square” and you might want them both.

To create the button above this code should in theory be enough:

....
property int dpi: Screen.pixelDensity*25.4 
....
BorderImage {
   id: scalableElement2
   source: "./image/Imagebase.png"
   width: 1.5*dpi; height: 0.36*dpi
   border.left: 25; border.top: 25
   border.right: 25; border.bottom: 25
  }

This code exposes the usual method we will use the most, ppi or dpi as the main metric.  it creates a button 1.5 inches long by 0.36 inches tall no mater what screen you use. However it still is not looking the same across multiple screens the curvatures on the corners will decrease as the PPI grows and the drop-shadow will become increasingly invisible. So one needs to introduce a similar solution that we created for the icons, were we use multiple baseImages for multiple PPI ranges. In my next blog post I will try to show how it can be done in an hopefully not too “ugly” way. We will also discuss the limitations of this method…

The post In Pixels we trust, Scalable UIs In QML part 2. appeared first on ACuteDesign.

A User Experience is not defined to exist in a single form factor any more.

It is expected that it can flow from one device to another seemingly adapting itself to each interaction paradigm, and yet to retain branding, mood and overall look and feel across multiple platforms.

On the other end, it’s expected that Designers and Developers create User Interface solutions that feel native across platforms and device types. At the same time a single platform can have a huge amount of variable interface possibilities that need to be taken into account by the UX/UI…

These are just a few problems one needs to take into account when designing apps for cross-platform deployments. At first sight it might seem like an impossible task to tackle.
In the next series of Posts I will share my experience in trying to square the circle, on what I have learned and how it can be implemented in QML…

Scalability.

Unfortunately there is no magic “design-once, deploy-everywhere” that can turn a windows 8 desktop application into an iOS touch app and also an embedded device UI. It’s not as  simple as scaling an app from a 4 inch screen into a 27 inch one by stating the number of pixels in the x and y dimensions (these days it can be the same). Users have different interaction expectations on different devices.

Scaling the same application on the same platform

Let’s start with what could be considered at first sight the easiest task… Making one application for say a mobile platform, for the sake of example, let’s use Android…
So in theory and all things being perfect you would design/implement an UI and you would expect it to work across the entire platform, on all devices, and in a way it does. You can stick with using only simple QtQuick.Controls elements that look native and scale nicely already; but there is a problem here, the number of elements available is limited, and the designer in you wants to mimic a certain intellectual concept into a visual metaphor that helps the user to “learn” the intended UI, and for that you need to design your special ButtonFoo or SpinSliderMagic etc, etc…
This is where QML is perfect, it was designed exactly for this, an easy way to declare such custom components that can mimic so much better the intended user interaction paradigm.

You can think of it like this: you have to paint a landscape and you have to do it with just Lego, there will be a moment when you just need a “triangular” non-existing piece to make the whole thing perfect. Because it’s the “triangular” thing that you can share across multiple platforms and unites the entire application across the diverse platforms, or it’s the triangular piece that “just works” for the content/interaction space ratio, whatever the problems you may have, you will need many “triangles”. QML allows you to do just that.

The Problem

In the desktop world, most screens have, up until recently, rather similar ppi’s (pixels per inch)
and the differences were many times mitigated by the distance to the user eyes. Larger screens had smaller ppi’s but were further away from the user eyes. This meant that the fundamental metric to design anything was the PIXEL – the smallest almost indivisible element that you could use. Because of this fact, fonts that apparently used different metrics were in reality using pixels – the logical value for ppi was hard-coded in the several desktop OSs. To avoid that, the font size – widget size ratio would change from screen A to screen B.
All of this was only possible due to the fact the range in ppi’s was rather limited…

Now let’s take a look at ppi’s on Android devices. It is totally possible to enter a store and look at 2 devices that look similar with radically different ppi’s from the 90s ish (where you can clearly discern individual pixels), to the mind-blowing 500s where it’s much more than the human eye can tell apart.
Yet one expects that a given app looks similar enough on both devices.

Using the same approach to element creation that we have on desktop simply does not work here.

If one uses a simple QML file that tries to scale a button-like element to the 2 screens above using a BorderImage element and simply states the distances in pixels, it will result in severely different looking apps to the user.

There are multiple ways to solve this issue and it is my plan to talk about them in the following posts, as, in my opinion, different methods can be used depending on the specific problem. However, to do that we need to expose and understand those very same problems, both from a conceptual as well as from a rendering pixel point of view.

The Usual method

The most common way to solve this issue is via the shipment of several different source images that are targeted at ppi ranges combined with ppi range metrics.

How one can implement such solutions, in current QML? What are the limitations? Where does it work the best?  All of this will be the subject of the next post.

So stay tuned as this might be a fairly lengthy subject…

The post Scalable UIs In QML appeared first on ACuteDesign.