Stately, a Kotlin Multiplatform Library

Stately, a Kotlin Multiplatform Library

This started as a monster single post, now split in 2. Part 1, Saner Concurrency, is about what Kotlin is doing with concurrency.

During my talk at KotlinConf, I promised a part 2 of Stranger Threads to better explain threading and state in Kotlin/Native. I built a library instead.

Update!!! Video and Slides from my Droidcon UK talk!

What is Stately

Stately is a collection of structures and utilities designed to facilitate Kotlin/Native and multiplatform concurrency. As of today, it is a set of expect/actual definitions that most apps will wind up needing, and a set of frozen, sharable collection classes that allow you to maintain mutable collections across threads.

Why does it exist?

Kotlin/Native, and hopefully soon, all of Kotlin, will be implementing Saner Concurrency. Native, today, has runtime rules, and notably a lack of “standard” concurrency primitives, that help ensure runtime concurrency sanity. The 2 basic rules are:

  1. All mutable state is in one thread. It can be transferred, but is “owned” by one thread at a time
  2. All shared state is immutable (frozen)

These are good rules, because they will make concurrent code safer.

However, there are times where being able to access shared, mutable state is pretty useful. That may change as we adapt architectural thinking, but at least for today, there have been a few tricky situations to arise without that available. Global service objects and caches, for example.

Kotlin/Native has a set of Atomic classes that allow you to mutate state inside of an immutable (frozen) object. The Stately collections are constructed with atomics.

For the most part, I’d expect Stately to be used sparingly, if at all, but it’s one of those things that you’ll really miss if you need it and don’t have it.

What you shouldn’t use it for

Kotlin/Native has rules that help encourage safer concurrency. Changing frozen state is, on some level, enabling manually managed concurrency. There are practical reasons to have these collections available, but if you’re using them a lot, it might be better to try for some architectural changes.

If you’re new to Native and running into mutability exceptions, you might be tempted to make everything atomic. This is equivalent to marking everything synchronized in Java.

It’s important to understand how Native’s threading and state work, and why you need a concurrent collection. But, you know, no judgements.

Basic Principles

The collections mostly act like their mutable counterparts. You designate generic types, and get/set/remove data entries. One key thing to note.

Anything you put into the collection will get frozen.

That is very important to understand. The collection itself is “mutable” in the sense that you can add and remove values, but the collection and values it holds are all frozen.

For data objects, this is generally OK, but for callbacks, this may be somewhat mind bending to understand. That’s not a Stately problem so much as a Kotlin/Native problem.

I’ve talked to several people who struggle with this reality. All I can say is it seems weird at first, but is not as big of a deal as you think. You just really need to be aware of what that means.

For example, in the Droidcon app, we’re using LiveData to build a reactive architecture. To avoid freezing everything in the Presenter/UI layer, we simply keep all of our callbacks thread local to the UI thread, which means they don’t need to be frozen. The lower level callbacks all exist in Sqldelight, and are frozen, but they exist just to push data back to the main thread and don’t capture UI-layer state.

Summary, it’s different, but not that bad. The details would turn this blog post back into a monster, so I’ll just push out my promised “Stranger Threads” deadline out by a couple weeks.

A lot of how we think about architecture will change as coroutines mature on Native, and as the community has some time to think about the implications. I suspect there will be less use for shared collections as that happens, but for today, they’re pretty useful.

Available Collections

CopyOnWriteList

Similar to Java’s CopyOnWriteArrayList. In fact, the JVM implementation is Java’s CopyOnWriteArrayList. Useful in cases with infrequent changes and critical read stability. Registering callback listeners, for example. If you’re changing the list often, or it’s large, each edit requires a full copy. Something to keep in mind.

SharedLinkedList

This is a mutable list that will have reasonable performance. All edits lock, just fyi, but individual edits themselves are pretty small, so locks are quick. You can hold onto a node reference that will allow you to swap or remove values without traversing the list, which is important in some cases.

There are 2 basic flavors. One has unstable but performant iterations, the other I’d call CopyOnIterateList. It’s similar to CopyOnWriteList, except the copy happens when you call iterate. This may prove more generally useful than COWAL, as it should handle frequent changes better, but if you’re editing often and iterating often, remember you’re dealing with an aggressively locking data structure.

The other flavor will let you iterate without copying or locks, but edits to the list will be reflected while you’re iterating. The edits are atomic, so you won’t get errors, but you’ll also wind up with potentially fuzzy iterations. If that isn’t an issue, this should be a better choice.

I am aware that there are lockless implementations of linked list, but I didn’t attempt one. Going for simple.

SharedHashMap

Implements a pretty basic version of our favorite structure HashMap. Performance should have similar characteristics to what we’re used to from Java🌶, although obviously locking will produce different absolute numbers. In short, same big O.

I probably don’t need to go into where a hash map would be useful, but in the service object context, it’s probably caches, which leads to…

SharedLruCache

Shared version of a least recently used cache. If you’re not familiar, there’s a cap on the number of values, and the “oldest” get bounced first. “Oldest” being defined as the least recently accessed.

Status

This is pretty new. As Kotlin 1.3 matures, we should have a more stable deployment, and may add some other features. I’ll try to add some issues for “help-wanted” in case anybody wants to contribute.

Also, as of today (Friday 10/26) the JS code has an implementation but tests need to be wired in. That means don’t use the JS until that happens.

Notes
The only supported Native implementation is for mac and iOS. Other Native targets should work, except we’ll need to find a lock implementation. Pthread_mutex is fine, except you need to destroy it explicitly, and K/N has no destructor. That means a ‘close’ method on the collection, which I’d rather avoid. Right now there’s a spin lock, but not sure if that’s a great idea.

The collection implementations could be described as minimal. My main goal was to start replacing some of the custom C++ code we’d put into earlier K/N implementations, which generally existed because of a need for shared state. If there’s a real need for something else, open an issue to discuss.

🌶 Similar to Java 7 and below. That is, if your bucket size doesn’t increase, as your entry size increases, or you have a bad hash, you start to approach N, because the bucket list means a lot of scanning. In Java 8, after the list gets to size 8(ish), it’ll store those values as a tree, so worst case gets capped. Our hash map resizes like Java’s, so you should only find yourself in “worst case” if your hash is bad, or if you have horrible settings on your map. You probably don’t need to know this, but Java 8’s optimization is kind of cool, and I didn’t bother trying to implement it 🙂

Saner Concurrency (and the cost of change)

Saner Concurrency (and the cost of change)

I’ve been talking about Kotlin/Native for a little while now, and one of the big things I tell people is that threading will be very different than what they’re used to. It’s just something that you need to learn and understand.

Kotlinconf was a couple weeks ago. Progress on Multiplatform was definitely a big topic. A more general discussion surfaced around how Jetbrains and the Kotlin team are thinking about concurrency. Andrey said in his Keynote that future Kotlin will focus on Saner Concurrency. I think we should unpack this a bit.

It is generally accepted that managing concurrency by using mutexes, synchronized blocks, etc (AKA. “manually” or “status quo”), is difficult and error prone. Concurrency in Java, with Java’s concurrency primitives, is difficult.

Some languages attempt to improve upon this situation using various approaches. Another point from the keynote I thought was great is that being unique is not important. Assimilating the best ideas of other languages is important (2:40).

Saner Concurrency

Kotlin/Native has baked into it some ideas about concurrency that make managing it safer. Andrey alluded to the idea that Kotlin, across variants, will start attempting to tackle “Saner Concurrency”. This is good to watch (37:00) to get a sense of where Kotlin is going, which makes me hopeful that not only will the various incarnations of Kotlin share the same concurrency concepts, but that those concepts will start to emerge as first class parts of the language at compile time rather than strictly runtime (Rust-like?).

By “various incarnations” I mean Kotlin-JVM will hopefully start getting some of the “new stuff” Kotlin/Native is getting🚧.

One of the consequences of change is that things will be different. We’ll need to learn new ideas, build different libraries, and adopt different ways of coding. That inevitably means resistance and time.

I wound up on the closing panel of the conference (thanks Hadi!), and one of the questions I gave an answer to addresses this issue. It wasn’t actually a question so much as expressing concern that the different threading model will slow down adoption (at 18:00)

I jumped in because, frankly, I had the same concerns earlier on. A few months ago I drafted up a blog post talking about how such a departure will impact adoption and may ultimately derail the effort. Thankfully, like most of my blog posts, I never finished it, and I don’t think that way anymore.

Considering it was on stage, at the end of a conference, I think I expressed my thoughts OK, but let me expand a bit here.

At the beginning of my talk, I told everybody about how I got there. I’d spent a couple years working on an architectural framework based on J2objc called Doppl. In short, compatible Android structures, lots of familiar libraries, and a gradle tool. Replicating testable mobile architecture across Android and iOS, in a familiar and practical manner.

How practical? The day after Google announced room, I got Room working on iOS, from the java source jars, in about 90 minutes.

Here’s the thing, though. Nobody cared. I bet more people read this post in the first week than read that post after a year and a half.

Obviously these aren’t direct parallels. The Android community wants Kotlin and iOS wants Swift, so pitching a Java/Objc thing isn’t fun (trust me). Community interest is critical, no matter how practical your pitch may be.

However, I think it’s also important to have a vision. Kotlin wants to adopt better, safer concurrency practices. They’re making the language available on many platforms, including Webassembly, and making “tough love” choices for a better future platform.

I think building something practical but somewhat compromised will make it easier to learn, and definitely easier to develop the library ecosystem necessary to support what currently exists, but it’s also uninspiring. It does not encourage change.

In short, yes, I think “giving up” on change would make adoption quicker for an individual iOS developer, but I don’t think it would be appealing to the iOS developer community. Change and new ideas, although they will take more work, are appealing and inspiring, and will ultimately be the best bet at gaining adoption.

When somebody says, “Kotlin is just Swift for Android”, this is meaningful pushback. It’s not simply “another language”.

In any case, it doesn’t matter. They’re doing it ✈️

This post started as one post, but turned into a monster. Part 2 will be released after my Droidcon UK talk Friday where we announce a new Kotlin Multiplatform state management library…

🚧 I didn’t want to derail up top, but one major issue for right now is that JVM and Native do some runtime things fairly differently. That is likely to create confusion early on, especially with teams that have engineers fully on one platform or the other. It would be great if we could run JVM with the same runtime checking as Native, even if just in a debug configuration.

Future of Shared Code = Kotlin Multiplatform

Future of Shared Code = Kotlin Multiplatform

If you’ve been following our development of Doppl for the last couple years, we’ve built a Gradle toolset for J2objc, an architectural structure with an Android compatible API, and published a number of popular libraries for that ecosystem. In other words, a full toolset extension of J2objc which allows sharing logic and architecture from Android to iOS.

J2objc is a mature, stable technology. If you have an iPhone, you’re more likely to have some J2objc in there than all the other non-HTML frameworks put together. J2objc makes sense not because it lets Java developers publish iOS code. It makes sense because it allows a platform native language produce platform native libraries, with optional code sharing, without any significant barrier or translation layer.

We use this framework with current clients, and would absolutely take new projects with it if we have the bandwidth to do so.

However, we’re officially out of the Doppl business.

The future of shared code and architecture, as far as we’re concerned, is Kotlin Multiplatform.

Some of the reasons we like it are the same as why we like J2objc. Native language, optional sharing, platform compatibility. As of the last few months, all R&D and open source development has been focused on KMP. As of KotlinConf this year (or shortly after) the tooling and libraries will be ready for production work. It is time to burn the ships.

To be better open source citizens, and to avoid internal distractions, we’ll be unpublishing Doppl and related libraries. If you’re one of the handful of orgs who use the library, please reach out and we’ll make sure you can access the repos.

I want to be clear that this doesn’t mean I think J2objc is bad tech. I still think it’s amazing technology. If you wanted to maximize your shared mobile architecture today, it would be significantly more productive than other options.

However, the community excitement around Kotlin Multiplatform is intense.

Over the next several months there will be a number of library announcements. It’s also not just for Android/iOS logic, but back end and front end architecture, including native mobile, web with JS, and webassembly. There is a lot of work to be done, but there’s a much wider scope of problems to solve.

We are, of course, looking for products and orgs interested in getting started. Also, I’m starting the search for our first Multiplatform specific engineer (besides me). Reach out.

Kotlin Multiplatform in the App Store!

Kotlin Multiplatform in the App Store!

Elf

I don’t know if this is the first Kotlin Multiplatform app in the iOS store, but it’s our first Kotlin Multiplatform app in the iOS store!

Technically we updated the existing Droidcon NYC/SF app, so the approval process was easier than a new app would be, but approved it was!

To read more about the app check out the other blog post

I most recently gave a talk about this on Thursday (8/15/2018) at Android Summit. I did my first experiment with recording my own talk, which went OK I think…

I’ll be doing a few more talks through the remainder of 2018. I’m most excited about KotlinConf, but will be doing a few other events as well.

Future Plans

We still need to release the Android app. Sometime in the next couple days.

The underlying libraries are being updated to Kotlin 1.3 eap as well as the latest Kotlin/Native builds. That should also be early this week.

That will probably be it till after Droidcon NYC. Speaking of, today is the last day for standard price tickets (use code ‘kotlinmp’ for a discount)!

App Size

Just a quick note about app size. We’ll write more about this later, but the summary is Kotlin/Native does really well with regards to download and install size vs other frameworks. The download size is under 10m. Our last version with J2objc was just under 30m. It had more images in it, but still. By comparison, we did the flutter sample a few months back, which came out to 20m. React Native kind of beat everybody, but anyway. Will need to make some bare-bones apps and do some comparisons.

Reach Out

As always, Touchlab is looking for orgs that want to get started with this tech. Reach out.

 

Droidcon NYC App!

Droidcon NYC App!

Mobile conference apps are an interesting beast. It’s a place where you can add the “weird” new tech people are talking about, and open source it, but that you’re not *quite* putting into production yet.

Fun fact. Our original Droidcon NYC app was where we first tried Kotlin, way back in 2014. For 2018, we have come full circle and ported the whole thing to Kotlin Multiplatform.

https://github.com/touchlab/DroidconKotlin/

There is an Android version and an iOS version. The UI’s are coded on their native platform, with Kotin on Android and Swift on iOS. The logic and much of the architecture is shared.

Building the app and talking architecture

The live code video from Friday (8/3/18) with Natalie…

Another fun fact. The conference schedule is not up on the site, but it IS in the app. If you absolutely need to see the Droidcon NYC 2018 schedule before next week, build the app.

There’s still some polishing left to be done before putting them in the app stores, but they’re functional now, and we wanted to get a fully functional mobile app out in the world for people to play around with. We might have to repeat the Sad Puppy mod before publishing to the Apple store, but the Android deploy doesn’t always go smoothly either. Cross fingers!

Libraries

The more interesting stack in the app is KNarch.db, Sqldelight, to LiveData. It’s a multiplatform, reactive architecture.

You *can* use the LiveData implementation in your app, but it’s largely for demonstration purposes. Something more useful and robust would need to be built (perhaps something like Reagent). For now, though, you can observe SQLDelight queries and update the UI as data is modified.

We’re also using Multiplatform Settings, which is an elegant multiplatform implementation of a properties store. Not doing a lot with it yet, but the plan is to add more properties.

Both SQLDelight and settings are published from forks we made. SQLDelight had some changes to support Kotlin/Native. For the MP library, it was purely to support Gradle dependencies.

Threads

The first thing you’ll notice, and I’m sure comment on, is the lack of coroutines. Coroutines on native have taken longer to emerge, and while building the app, they weren’t super ready. I’ve seen some progress being made while wrapping up the app, but Kotlin/Native is changing fast and trying to include all the stuff that they add while building is a good way to kill your productivity. High on the list of todos is investigating the state of coroutines and seeing what we can do there.

For now, though, how threading works. There are a couple functions that power the explicit backgrounding in the app. Look at

sessionize/src/main/kotlin/co/touchlab/sessionize/utils/Functions.kt 

expect fun <B> backgroundTask(backJob:()-> B, mainJob:(B) -> Unit)
expect fun backgroundTask(backJob:()->Unit)
expect fun networkBackgroundTask(backJob:()->Unit)

In the JVM these get sent to an ExecutorService. In K/N, these get run on a worker. The interesting trick is how the ‘mainJob’ returns. That should be its own blog post, but if you understand K/N, ‘mainJob’ is kept in a ThreadLocal set in the main thread, so there’s no need to freeze it. If you don’t know what I’m talking about, we should be putting out some followup K/N threading posts.

What’s ‘networkBackgroundTask’? Just a separate queue. The networking stuff generally takes longer.

LiveData

There’s kind of a debate in the Android world between RxJava and LiveData. I won’t comment on that. We picked LiveData because we needed to implement it on iOS, and LiveData is very, very small. RxJava is not. That’s it.

SQLDelight needed some internal mods to support query listeners that could be frozen and exist in multiple threads. LiveData uses them and will push to the main thread when data is updated. You have to explicitly detach from the query because of a frustrating multiplatform issue. Please vote to have that one looked into.

Packaging

There seems to be a lot of movement around the K/N Gradle plugin, to bring it more in line with the others. The dependency mechanism is likely to change, so we’re in kind of an unknown space. In this app and with the libraries, we’re using the older plugin and dependency mechanism. It’ll look strange to Java folks, but it works fine. Our situation is somewhat complicated by the fact that KNarch.db and threads both have some C++ compiled and included. Personally, I’d like to see the K/N tools embrace their C++ underpinnings and make C++ easy to include in your own projects. If you’re going to get serious about K/N development you should at least be familiar with the runtime folder of K/N. Yada yada, dependencies, and packaging will likely change over the next few months.

Testing

None. Looking for hands, if anybody wants to play!

Future

Will try to include http-client-common. I ran into issues out of the gate, and the app is only doing GET calls, so I just hand rolled them.

As mentioned, investigate coroutines.

K/N 8.1 has been released. I think we’d want to re-publish all the libs as well as the project itself, so I’m holding off on that.

Looking for

If you’d like to play with Kotlin multiplatform and push some code back, we’ll be looking for a few things.

The UI on both platforms is partially Material Components. The goal was to have a relatively homogenous style definition so any conference using sessionize could use the app (cough). It’s not quite there. I did much of the app work as the rest of the team is doing actual client work, and my iOS UI understanding is minimal at best. Cleaning and consolidating that would be great.

Testing, as mentioned. This app, as much as anything, is an example of how to build Kotlin Multiplatform on mobile, and testing is going to be critical to selling that idea. By that I mean the tests don’t necessarily have to be super useful. They just need to exist.

Maybe some design adjustment. We largely took what we had in previous years on both platforms.

Status

It’s still a little early to be producing apps, but things are moving fast. I’ve been predicting that we’ll see several libraries emerge over the summer, and by late 2018/early 2019 there will be the full library stack and toolset necessary to be fully productive. I’m betting that’s still true. Hopefully, anyway, because I’ve pitched a lot of talks around it: http://androidsummit.org/, https://www.nyc.droidcon.com/, https://kotlinconf.com/, and at least 2 more that I’m still confirming.

To be clear, it’s a little early if you don’t know what you’re doing, but that’s when you bring in your friends at:

We are seriously looking for orgs that want to get started early on realizing shared architecture between mobile platforms as well as the web. Get in touch.