Saner Concurrency (and the cost of change)

Saner Concurrency (and the cost of change)

I’ve been talking about Kotlin/Native for a little while now, and one of the big things I tell people is that threading will be very different than what they’re used to. It’s just something that you need to learn and understand.

Kotlinconf was a couple weeks ago. Progress on Multiplatform was definitely a big topic. A more general discussion surfaced around how Jetbrains and the Kotlin team are thinking about concurrency. Andrey said in his Keynote that future Kotlin will focus on Saner Concurrency. I think we should unpack this a bit.

It is generally accepted that managing concurrency by using mutexes, synchronized blocks, etc (AKA. “manually” or “status quo”), is difficult and error prone. Concurrency in Java, with Java’s concurrency primitives, is difficult.

Some languages attempt to improve upon this situation using various approaches. Another point from the keynote I thought was great is that being unique is not important. Assimilating the best ideas of other languages is important (2:40).

Saner Concurrency

Kotlin/Native has baked into it some ideas about concurrency that make managing it safer. Andrey alluded to the idea that Kotlin, across variants, will start attempting to tackle “Saner Concurrency”. This is good to watch (37:00) to get a sense of where Kotlin is going, which makes me hopeful that not only will the various incarnations of Kotlin share the same concurrency concepts, but that those concepts will start to emerge as first class parts of the language at compile time rather than strictly runtime (Rust-like?).

By “various incarnations” I mean Kotlin-JVM will hopefully start getting some of the “new stuff” Kotlin/Native is getting🚧.

One of the consequences of change is that things will be different. We’ll need to learn new ideas, build different libraries, and adopt different ways of coding. That inevitably means resistance and time.

I wound up on the closing panel of the conference (thanks Hadi!), and one of the questions I gave an answer to addresses this issue. It wasn’t actually a question so much as expressing concern that the different threading model will slow down adoption (at 18:00)

I jumped in because, frankly, I had the same concerns earlier on. A few months ago I drafted up a blog post talking about how such a departure will impact adoption and may ultimately derail the effort. Thankfully, like most of my blog posts, I never finished it, and I don’t think that way anymore.

Considering it was on stage, at the end of a conference, I think I expressed my thoughts OK, but let me expand a bit here.

At the beginning of my talk, I told everybody about how I got there. I’d spent a couple years working on an architectural framework based on J2objc called Doppl. In short, compatible Android structures, lots of familiar libraries, and a gradle tool. Replicating testable mobile architecture across Android and iOS, in a familiar and practical manner.

How practical? The day after Google announced room, I got Room working on iOS, from the java source jars, in about 90 minutes.

Here’s the thing, though. Nobody cared. I bet more people read this post in the first week than read that post after a year and a half.

Obviously these aren’t direct parallels. The Android community wants Kotlin and iOS wants Swift, so pitching a Java/Objc thing isn’t fun (trust me). Community interest is critical, no matter how practical your pitch may be.

However, I think it’s also important to have a vision. Kotlin wants to adopt better, safer concurrency practices. They’re making the language available on many platforms, including Webassembly, and making “tough love” choices for a better future platform.

I think building something practical but somewhat compromised will make it easier to learn, and definitely easier to develop the library ecosystem necessary to support what currently exists, but it’s also uninspiring. It does not encourage change.

In short, yes, I think “giving up” on change would make adoption quicker for an individual iOS developer, but I don’t think it would be appealing to the iOS developer community. Change and new ideas, although they will take more work, are appealing and inspiring, and will ultimately be the best bet at gaining adoption.

When somebody says, “Kotlin is just Swift for Android”, this is meaningful pushback. It’s not simply “another language”.

In any case, it doesn’t matter. They’re doing it ✈️

This post started as one post, but turned into a monster. Part 2 will be released after my Droidcon UK talk Friday where we announce a new Kotlin Multiplatform state management library…

🚧 I didn’t want to derail up top, but one major issue for right now is that JVM and Native do some runtime things fairly differently. That is likely to create confusion early on, especially with teams that have engineers fully on one platform or the other. It would be great if we could run JVM with the same runtime checking as Native, even if just in a debug configuration.

Future of Shared Code = Kotlin Multiplatform

Future of Shared Code = Kotlin Multiplatform

If you’ve been following our development of Doppl for the last couple years, we’ve built a Gradle toolset for J2objc, an architectural structure with an Android compatible API, and published a number of popular libraries for that ecosystem. In other words, a full toolset extension of J2objc which allows sharing logic and architecture from Android to iOS.

J2objc is a mature, stable technology. If you have an iPhone, you’re more likely to have some J2objc in there than all the other non-HTML frameworks put together. J2objc makes sense not because it lets Java developers publish iOS code. It makes sense because it allows a platform native language produce platform native libraries, with optional code sharing, without any significant barrier or translation layer.

We use this framework with current clients, and would absolutely take new projects with it if we have the bandwidth to do so.

However, we’re officially out of the Doppl business.

The future of shared code and architecture, as far as we’re concerned, is Kotlin Multiplatform.

Some of the reasons we like it are the same as why we like J2objc. Native language, optional sharing, platform compatibility. As of the last few months, all R&D and open source development has been focused on KMP. As of KotlinConf this year (or shortly after) the tooling and libraries will be ready for production work. It is time to burn the ships.

To be better open source citizens, and to avoid internal distractions, we’ll be unpublishing Doppl and related libraries. If you’re one of the handful of orgs who use the library, please reach out and we’ll make sure you can access the repos.

I want to be clear that this doesn’t mean I think J2objc is bad tech. I still think it’s amazing technology. If you wanted to maximize your shared mobile architecture today, it would be significantly more productive than other options.

However, the community excitement around Kotlin Multiplatform is intense.

Over the next several months there will be a number of library announcements. It’s also not just for Android/iOS logic, but back end and front end architecture, including native mobile, web with JS, and webassembly. There is a lot of work to be done, but there’s a much wider scope of problems to solve.

We are, of course, looking for products and orgs interested in getting started. Also, I’m starting the search for our first Multiplatform specific engineer (besides me). Reach out.

Kotlin Multiplatform in the App Store!

Kotlin Multiplatform in the App Store!

Elf

I don’t know if this is the first Kotlin Multiplatform app in the iOS store, but it’s our first Kotlin Multiplatform app in the iOS store!

Technically we updated the existing Droidcon NYC/SF app, so the approval process was easier than a new app would be, but approved it was!

To read more about the app check out the other blog post

I most recently gave a talk about this on Thursday (8/15/2018) at Android Summit. I did my first experiment with recording my own talk, which went OK I think…

I’ll be doing a few more talks through the remainder of 2018. I’m most excited about KotlinConf, but will be doing a few other events as well.

Future Plans

We still need to release the Android app. Sometime in the next couple days.

The underlying libraries are being updated to Kotlin 1.3 eap as well as the latest Kotlin/Native builds. That should also be early this week.

That will probably be it till after Droidcon NYC. Speaking of, today is the last day for standard price tickets (use code ‘kotlinmp’ for a discount)!

App Size

Just a quick note about app size. We’ll write more about this later, but the summary is Kotlin/Native does really well with regards to download and install size vs other frameworks. The download size is under 10m. Our last version with J2objc was just under 30m. It had more images in it, but still. By comparison, we did the flutter sample a few months back, which came out to 20m. React Native kind of beat everybody, but anyway. Will need to make some bare-bones apps and do some comparisons.

Reach Out

As always, Touchlab is looking for orgs that want to get started with this tech. Reach out.

 

Droidcon NYC App!

Droidcon NYC App!

Mobile conference apps are an interesting beast. It’s a place where you can add the “weird” new tech people are talking about, and open source it, but that you’re not *quite* putting into production yet.

Fun fact. Our original Droidcon NYC app was where we first tried Kotlin, way back in 2014. For 2018, we have come full circle and ported the whole thing to Kotlin Multiplatform.

https://github.com/touchlab/DroidconKotlin/

There is an Android version and an iOS version. The UI’s are coded on their native platform, with Kotin on Android and Swift on iOS. The logic and much of the architecture is shared.

Building the app and talking architecture

The live code video from Friday (8/3/18) with Natalie…

Another fun fact. The conference schedule is not up on the site, but it IS in the app. If you absolutely need to see the Droidcon NYC 2018 schedule before next week, build the app.

There’s still some polishing left to be done before putting them in the app stores, but they’re functional now, and we wanted to get a fully functional mobile app out in the world for people to play around with. We might have to repeat the Sad Puppy mod before publishing to the Apple store, but the Android deploy doesn’t always go smoothly either. Cross fingers!

Libraries

The more interesting stack in the app is KNarch.db, Sqldelight, to LiveData. It’s a multiplatform, reactive architecture.

You *can* use the LiveData implementation in your app, but it’s largely for demonstration purposes. Something more useful and robust would need to be built (perhaps something like Reagent). For now, though, you can observe SQLDelight queries and update the UI as data is modified.

We’re also using Multiplatform Settings, which is an elegant multiplatform implementation of a properties store. Not doing a lot with it yet, but the plan is to add more properties.

Both SQLDelight and settings are published from forks we made. SQLDelight had some changes to support Kotlin/Native. For the MP library, it was purely to support Gradle dependencies.

Threads

The first thing you’ll notice, and I’m sure comment on, is the lack of coroutines. Coroutines on native have taken longer to emerge, and while building the app, they weren’t super ready. I’ve seen some progress being made while wrapping up the app, but Kotlin/Native is changing fast and trying to include all the stuff that they add while building is a good way to kill your productivity. High on the list of todos is investigating the state of coroutines and seeing what we can do there.

For now, though, how threading works. There are a couple functions that power the explicit backgrounding in the app. Look at

sessionize/src/main/kotlin/co/touchlab/sessionize/utils/Functions.kt 

expect fun <B> backgroundTask(backJob:()-> B, mainJob:(B) -> Unit)
expect fun backgroundTask(backJob:()->Unit)
expect fun networkBackgroundTask(backJob:()->Unit)

In the JVM these get sent to an ExecutorService. In K/N, these get run on a worker. The interesting trick is how the ‘mainJob’ returns. That should be its own blog post, but if you understand K/N, ‘mainJob’ is kept in a ThreadLocal set in the main thread, so there’s no need to freeze it. If you don’t know what I’m talking about, we should be putting out some followup K/N threading posts.

What’s ‘networkBackgroundTask’? Just a separate queue. The networking stuff generally takes longer.

LiveData

There’s kind of a debate in the Android world between RxJava and LiveData. I won’t comment on that. We picked LiveData because we needed to implement it on iOS, and LiveData is very, very small. RxJava is not. That’s it.

SQLDelight needed some internal mods to support query listeners that could be frozen and exist in multiple threads. LiveData uses them and will push to the main thread when data is updated. You have to explicitly detach from the query because of a frustrating multiplatform issue. Please vote to have that one looked into.

Packaging

There seems to be a lot of movement around the K/N Gradle plugin, to bring it more in line with the others. The dependency mechanism is likely to change, so we’re in kind of an unknown space. In this app and with the libraries, we’re using the older plugin and dependency mechanism. It’ll look strange to Java folks, but it works fine. Our situation is somewhat complicated by the fact that KNarch.db and threads both have some C++ compiled and included. Personally, I’d like to see the K/N tools embrace their C++ underpinnings and make C++ easy to include in your own projects. If you’re going to get serious about K/N development you should at least be familiar with the runtime folder of K/N. Yada yada, dependencies, and packaging will likely change over the next few months.

Testing

None. Looking for hands, if anybody wants to play!

Future

Will try to include http-client-common. I ran into issues out of the gate, and the app is only doing GET calls, so I just hand rolled them.

As mentioned, investigate coroutines.

K/N 8.1 has been released. I think we’d want to re-publish all the libs as well as the project itself, so I’m holding off on that.

Looking for

If you’d like to play with Kotlin multiplatform and push some code back, we’ll be looking for a few things.

The UI on both platforms is partially Material Components. The goal was to have a relatively homogenous style definition so any conference using sessionize could use the app (cough). It’s not quite there. I did much of the app work as the rest of the team is doing actual client work, and my iOS UI understanding is minimal at best. Cleaning and consolidating that would be great.

Testing, as mentioned. This app, as much as anything, is an example of how to build Kotlin Multiplatform on mobile, and testing is going to be critical to selling that idea. By that I mean the tests don’t necessarily have to be super useful. They just need to exist.

Maybe some design adjustment. We largely took what we had in previous years on both platforms.

Status

It’s still a little early to be producing apps, but things are moving fast. I’ve been predicting that we’ll see several libraries emerge over the summer, and by late 2018/early 2019 there will be the full library stack and toolset necessary to be fully productive. I’m betting that’s still true. Hopefully, anyway, because I’ve pitched a lot of talks around it: http://androidsummit.org/, https://www.nyc.droidcon.com/, https://kotlinconf.com/, and at least 2 more that I’m still confirming.

To be clear, it’s a little early if you don’t know what you’re doing, but that’s when you bring in your friends at:

We are seriously looking for orgs that want to get started early on realizing shared architecture between mobile platforms as well as the web. Get in touch.

SQLite on Kotlin/Native

SQLite on Kotlin/Native

Tada! (Matty Matheson, Viceland, Giphy)

Today we’re releasing an early-ish version of Sqlite for K/N. More specifically, its a version of the Android Sqlite stack for K/N, that can be used in multiplatform projects.

 

Back in early May we demoed SQLite and SQLDelight running on iOS with Kotlin/Native and multiplatform. To implement the iOS side, I wrapped J2objc/Doppl code, which worked, but wasn’t exactly pretty under the hood.

In that post I said the goal was to get to this:

Pretty diagram

Well, that’s this.

I made a promise at the NYC meetups to release something before Droidcon Berlin. Flying out tomorrow, so here it is!

This is definitely an early release. Over the summer we’ll be updating the architecture, particularly around the multiplatform portion of it. Also, because K/N’s memory and threading model are significantly different than the JVM’s, some features were disabled and architecture simplified to be able to release something stable. We’ll be revisiting those decisions and adjusting the architecture as we go.

Details of the process and design will have to wait for follow on blog posts and/or talks. For now, here’s an overview of what’s available.

Main Project

The repo has 4 main folders.

cpp

This contains the code that directly talks to sqlite3. Much of this code is ported from AOSP and Android’s SQLite stack, but types are changed from JNI to what K/N uses. There’s also some code to facilitate state management and threading in K/N.

kotlin

The really interesting folder is kotlin-ios. This is the implementation of SQLite on iOS. You’ll notice a familiar set of classes (SQLiteDatabase, etc). The majority of these originate from AOSP, but have been updated to Kotlin and to live in an environment without the JVM or directly managed multithreading.

Much of the test folder is ported from CTS. This area of the code base will need a lot more TLC in the future, but the CTS tests provide a nice base to work from.

If you’re curious (and on a mac), run ‘./gradlew build’ on the kotlin folder, then look under kotlin-ios/build for the generated kexe file. This will run the test suite.

sample-notepad and sample-notepad-sqldelight

Two basic sample apps. One just SQL, one running SQLDelight’s multiplatform. Spoilers, the SQLDelight one is more interesting.

***Update!!!***
I totally forgot to mention that the android side of the samples isn’t done. Sqlite working on android isn’t exactly interesting. Soon, though.

SQLDelight

SQLDelight’s Kotlin generation and multiplatform support is currently in alpha. I have a fork with a driver implementation that allows it to function on iOS. I would not try to put this into production today, but definitely kick the tires, and something more official will certainly be coming along soon.

Usage

Currently there isn’t a dependency mechanism for K/N, as far as I know, so using the library means copying around the compiled artifacts. See the sample apps on how this is done. This gets super painful fast, so some kind of dependency publishing and management is high on the wish list.

Threading?

Threading on K/N is very different than what you’re probably used to in the JVM. The SQLite stack on Android does a lot of thread management internally. SQLiteOpenHelper, SQLiteDatabase, and all of its associated parts can (and need to be) frozen to be passed around to multiple threads. They all use the same sqlite connection under the hood. Internal state does change, but is kept in C++ structures in a way that is safe for K/N.

I tried to get clever with K/N and keep some non-frozen data that got passed around between threads. That ate a couple weeks and was sad. Would not recommend.

Cursor, I feel inclined to point out, is not thread safe, but Cursor isn’t really thread safe in the JVM either. I think it would be useful if K/N had a way to mark a class that specifically can’t, or shouldn’t, be frozen, such that it’ll throw at runtime if you try, but I guess you’ll find out soon enough. Anyway…

I’m way overdue for another Stranger Threads post. In it I’ll talk more about state and threads (and Atomics!).

KNarch?

K(otlin)N(ative)Arch(itecture). Sounds like narc. Names are hard. Suggestions welcome.

Want to help?

There’s a bunch of work that can be done here. Pick through issues, “needs hands” in particular. It’s a good opportunity to understand K/N at an architectural level. Video explaining how to set up the build environment coming after Droidcon Berlin.

Speaking of …

Other than Berlin, expect to hear more at Droidcon NYC, (probably) Droidcon London, and (very excited about this) KotlinConf! Also probably every Android NYC meetup for the foreseeable future. Sorry in advance, local friends.

Jobs!

Touchlab is currently working on projects with Doppl, and aggressively hunting down early Kotlin/Native adoption clients. We are very committed to delivering client work and open source code on this platform. If this is what you’d like to do, get in touch. We’re also remote friendly. Just FYI.

Projects!

If you’re part of an org that wants to start sharing architecture and be part of the cool crowd, we definitely want to talk to you.

Now, starting slides for the Berlin talk… (update: taking longer than planned)

Tick tock