Mobile Code Sharing with Kotlin Multiplatform

Mobile Code Sharing with Kotlin Multiplatform

Native mobile development for iOS and Android generally involves writing two (2) separate code bases, picking an “all in” cross platform tool, or using some form of code sharing to bridge the gap.

For long-term mobile developers, there is a sense of dread around any form of “cross-platform” due to generally negative experiences and outcomes. Then tooling and ecosystems are generally bad, the performance is often bad, etc.

There is nothing about the native tools that make them inherently better options, other than Apple and Google have huge incentives to make great apps for their platforms, and the teams that make the tools control the platforms. Other technology that intends to “keep up” needs to find its own incentives (and, by extension, funding), and needs to be very careful not to try to beat the native platforms at what they do best.

Native developers tend to talk about “cross-platform” as one concept, with all tech being rolled into “native” and “cross -platform”. In reality, all these options vary considerably and it’s important to define them appropriately so we can discuss which options are better for which circumstances.

The most concise definition of Kotlin Multiplatform I can give is as follows:

 

  • optional
  • natively integrated
  • open source
  • mobile code sharing platform
  • based on the popular, modern language Kotlin
  • facilitates non-UI logic availability on many platforms

 

Optional

Optional means the tech is designed to be used along with the native tech. Most cross-platform frameworks allow you to integrate them into existing apps, but are generally not designed to make that easy. “Possible” is not the same as “actively designed for”.

The optional sharing nature means it is easy to evaluate KMP in your existing apps. You can add code incrementally, and critically, if some shared code isn’t performing as expected, you can “just code it natively” without any major issues. The optional sharing nature makes KMP low-risk.

 

Natively Integrated

Other code sharing solutions, such as C++, JS and Rust, generally involve a bridging layer. C++ is “native”, but from the JVM you need to talk to it through JNI. That will have extra code and performance implications. The same goes for other code-sharing frameworks. They run in their own expected runtimes, with their own expected rules. Kotlin was designed to interop with the JVM directly, and Kotlin Native has been designed to use LLVM to produce actual “native” code.

For iOS, the Kotlin compiler outputs an Xcode Framework that Objective-C and Swift can communicate with directly. It looks like any other Objective-C framework, and with Swift ABI stability in 5.1, we can expect direct Swift interop next year.

Native integration dovetails with optional code sharing to reduce barriers to integration (easy to call, low-performance impact, etc) and reduce risk.

Open Source

Most new languages and platforms are developed as open source. It would be very difficult to create a closed source ecosystem in 2019. Not only does open source significantly reduce the risk of debugging issues, but it also allows the community to participate in the platform development.

To help ensure good public direction and guidance, Jetbrains and Google, along with other industry representatives, created the Kotlin Foundation to help steer future Kotlin direction.

Popular

This is important.

For libraries, support, training, hiring, etc. The ecosystem you pick needs to be popular. Having a very small ecosystem introduces it’s own form of risk. Kotlin Multiplatform is currently small, but growing very rapidly. It is very unlikely to lose momentum now. There will be an early adopter impact, but there’s also an early adopter opportunity.

For teams that want to be involved in open source, and to be able to recruit developers interested in new tech, it’s a great time to get involved.

Modern

Kotlin is a changing platform, attempting to incorporate lessons from other languages and ecosystems. This will mean better productivity and safety, and a longer “shelf life” for the ecosystem as a whole. It is a future-proofed ecosystem.

Mobile Code Sharing

It is important to distinguish “code sharing” from “mono-UI” and other cross-platform technologies.

One of my own favorite quotes is:

“The history of shared UI has a lot of pain and failure. The history of shared logic is the history of computers.”

Speaking of native mobile specifically, Android and iOS are very similar under the hood. They’re both Unix-y, have files, threads, networking, sqlite, etc. If the tooling was great, and the integration was seamless, it would be crazy not to create a shared codebase for at least some of the logic and architecture of applications. You can write it once, test it once, and only pay one cost for maintenance.

Code sharing is in contrast to shared UI, and “3rd platform” architectures. Picking Flutter, or React Native, or Xamarin (Ionic or any web thing, etc) means you will make a heavy investment in that platform. They don’t integrate well with the native ecosystems. They are what we call “Big Decisions”, and stress “Capital B, Big Decisions”. Big decisions are risky. They take effort to make. You need to train/hire for another 3rd platform. You need to recode a bunch, and/or write bridge code to the native code already in your app. You will likely need to fill in gaps that don’t exist. You need to accept the risk that the primary developer of the platform may decide to stop supporting it.

For that reason, it’s very important to understand the financial incentives involved. Most of these tools are free. If you aren’t the customer, you’re maybe the product. In any case, these frameworks and tools aren’t free to make or support, so why they’re made is an important consideration. As the risk increases, the investment and reliance on that tech should decrease, which is the opposite of how most cross platform options are architected (i.e. they want you to do everything in their tool, and take on more of that risk). 

As a side note, Kotlin is the preferred language for Android. In the case of native mobile, this makes Kotlin unique for code sharing. C++, JS, Rust, Go, etc can all be used, but none of them is the “preferred” language for either platform. In the case of KMP, there’s no “3rd” ecosystem. Again, going back to risk, not adding a 3rd language is important. Also, because of how Android the Operating System is created and deployed to actual devices, there tends to be a much wider range of versions and issues. Being able to avoid adding any risk to the Android side is a huge benefit.

JetBrains

The final important note is that Kotlin is made by JetBrains. They make some of the most popular tools for developers in a wide range of language ecosystems, but have absolutely dominated the JVM tooling world. Developer experience is critical for adoption, and if anybody else was trying to build a shared code solution at this scale, I’d say their chances aren’t great. The tooling is still in development, but if anybody can pull it off, it’s JetBrains.

JetBrains is a profitable company that sells developer tools to developers. They do not share the risk of venture-backed open source (Parse, Realm, etc). It is *possible* that they’ll sell to somebody, but there is no existential threat forcing them to do so. I started using their Java IDE back in 2000. That is ancient in the tech world. They have thrived using the old model of making and selling a product, and it’s likely they will continue doing just that.

Useful introduction to Kotlin Multiplatform resources 

Musings on the Current State of Kotlin Multiplatform (Q3 ’19)

Musings on the Current State of Kotlin Multiplatform (Q3 ’19)

I’ve spent a lot of time chatting in Slack and on Twitter about the state of Kotlin Multiplatform. These are usually piecemeal conversations but often center around common themes. To spread the word better and help streamline these conversations, I’m going to write periodic posts about the current state of the KMP ecosystem.

Catching Up

I think we can start by revisiting some predictions I had last year around KMP in 2019.

 

Mainstream Adoption

The overall prediction was that we’d be seeing recognizable names diving into KMP and releasing serious apps. Not exactly a wild guess. It is safe to say that was correct, although with a caveat. We’ve been having lots of conversations with teams evaluating and shipping KMP, but these are mostly private. Similarly, if you installed an app using KMP on iOS, you wouldn’t know it, which is the point. I expected tech companies to be a little more forward about the interesting tech they’re using, as I think it really helps with recruiting, but it turns out that not everybody thinks that way.

It is still early days, and you still need to know what you’re doing, but production code is certainly shipping with KMP in Q3.

Multithreaded Coroutines

Not yet. I was *way* off on that. Will discuss later.

Compiler Plugins

This is also “not yet”, although if you want something very specific you could persuade the Kotlin Native team to let you bake something in (if you’re looking for something serious to work on, mockk native). The Kotlin compiler in general is going through big changes. There’s a consolidation happening that will be good long term, but we’ll be waiting for official plugin support in the short.

Other Libraries 

I mean, sure.

Paid License Debugger 

This prediction is interesting, and meta for me. We released an Xcode pluginthat lets you debug Kotlin. Jetbrains has also released a version of App Code that can be used for debugging, and recently a native debugger for Intellij, although all of these tools will need some future polish.

Reactive Library

Yes, ish. I was thinking that would happen after Multithreaded coroutines. There is Reaktive. It is more of a “traditional” Rx-like library. I think most apps at this point use something built in-house.

Q3 2019

Rather than just a big rundown of where things are at, I’m going to create sort of a composite character and consolidate multiple conversations I’ve had recently.

Q: Is anybody currently using KMP in their apps? There have been a lot of blog posts from earlier in the year, but the situation seemed flakey. Has there been much progress since then?

A: Yes! People are using KMP in apps. KMP is relatively new, generating lots of excitement, and moving very fast. That means there’s been a lot of aging content. If getting started or looking for updated info, it’s best to check out the Kotlin Slack, check out some tutorials, and see more detailed docs. If you’re looking for a super quick intro, check out our getting started series. We should have a few more coming out soon. They’re quick, and we’re going to keep them updated with each release, so they don’t age too fast.

Q: We are exploring the possibility of using Kotlin MPP for business logic and architecture (storage, networking, etc). Is that feasible and how much logic can be shared?

A: That is really what KMP is designed to do, so it’s definitely feasible. It will not be easy to build UI’s, certainly not shared ones (although there will certainly be shared UI options in the future). What’s really missing at this point are some libraries you might want to use, like date/time support, or local file access. The good news is that 1) library work is accelerating as the community grows, and 2) Kotlin has an optional interop design, so it’s easy to plug in whatever you want from the native platform to implement a missing feature. If you can’t find a library for something you want to do, it’s easy to just code it “native”. Date formatting, for example.

Q: I’ve heard coroutines aren’t supported on iOS, and they’re needed for concurrency. How does that work?

A: Interesting topic. Coroutines in native (and JS) are single-threaded. That’s not the same as unsupported. You cannot, however, do all the stuff you would on JVM/Android. That has definitely put off some developers, but this is really multiple questions. Coroutines are not needed for concurrency. In the context of native mobile development, we usually want to do something off the main thread, then return to the main thread. You can do this, and it’s straightforward to write. We use suspend functions in the Droidcon app where the main thread suspends, a background task is run, then execution resumes in the main thread. Flow is more complex, but we’re using it in a Firestone SDK wrapper currently (not public yet). Ktor uses suspend functions instead of callbacks for network calls.

However, the whole community expects multithreaded coroutines will significantly increase adoption. There are no target dates, but there is ongoing work and I’d bet something to appear around Kotlinconf.

Understanding native concurrency helps the discussion. I have a couple of blog posts (and that’s what I’m talking about at Kotlinconf):

Q: What’s the Swift and general iOS interop situation like?

A: There are the mechanical interop details, then there’s what I’d call the Swift developer interop, which is related, but different. Swift interop with Kotlin is through an Objective-C interface generated by the Kotlin compiler. That means whatever issues a Swift developer will encounter with Swift to Objc interop will be present. On the other hand, Apple has spent considerable time trying to make that interop as painless as possible. That means Swift naming overrides, generics handling, etc. The Kotlin Objc output is constantly improving, and currently supports many of those features.

There are no details yet, but I have it on good authority that direct Swift interop may be a priority in the not-too-distant future.

The politics of shared code and Kotlin on iOS are a different story. The goal is to steadily improve the tooling experience for everybody, including iOS developers. That is certainly a huge goal of ours. That should help, but there’s going to need to be some selling on the idea too. Need to start seeing Kotlin talks as iOS meetups!

Q: Our iOS team is really excited about SwiftUI and said we should hold off on Kotlin until we figure that out.

A: Not in the form of a question, but I’ll rephrase: “Can you use Kotlin with SwiftUI?” Yes. Not the actual “UI” part, but the logic and architecture, which is what KMP is really for, sure. SwiftUI doesn’t change the equation much. We’re all pretty excited about it too.

You cannot have structs coming out of Kotlin. That is true. The data you use in SwiftUI does not need to be a struct. Struct support from Kotlin is an ongoing discussion, and we may implement something in the future that extends the interop, and/or down the road there may be formal support in Kotlin. For now, though, if the lack of structs is a show-stopper, then you’ll have an issue.

What’s Happening at Touchlab?

Touchlab is hiring full time and looking for contractors. Mostly Android, some multiplatform in the future: http://go.touchlab.co/trdw

We recently launched The Touchlab Refactory; an incremental refactoring service with KMP which focuses on a single feature at a time with no disruption to your current development workflow. Learn more here: http://go.touchlab.co/refactory

 
Kotlin Xcode Plugin

Kotlin Xcode Plugin

For native mobile developers using Kotlin Multiplatform, the iOS dev experience will be critical. Over the next several months at least, that will be the primary focus of Touchlab’s Kotlin R&D and our collaboration with Square.

Our first release is an Xcode Plugin for Kotlin. Once installed, Xcode will recognize Kotlin files as source files, provide keyword coloring, and most importantly, will let you interactively debug Kotlin code running in the iOS simulator.

 

 

Depending on project setup, that includes regular code as well as Kotlin unit test code. We’ll be providing info on how to run your unit tests in Xcode soon.

Download Plugin Archive Here

 

 

Live Code!

Setting up the plugin and Xcode can be a little tricky. We’ll be doing a live code session Friday 4/26 at 3pm EST. We’ll be setting up a project, demoing debug functionality, helping out some others with setup, and time permitting, checking out App Code as well. If you’d like to watch, sign up here.

If you’re planning on using the Xcode debugger and would like some live help, tick the box on the form above. You’ll need to share your screen, or at least have an open source project we can run. We’ll be picking 1 or 2 projects and attempting to configure them live.

 

App Code?

We are still very much fans of JetBrains tools and look forward to a fully integrated dev experience with AppCode. Check out the blog post about updated support in v2019.1. AppCode has started to ship with an interactive debugger, and we look forward to this product maturing along with Kotlin as a platform.

However, convincing iOS devs to give Kotlin MP a shot is easier with an Xcode option. This plugin is definitely not a full featured tool like IntelliJ IDEA and AppCode, but will allow iOS developers to step through and debug shared code.

Our thinking is that on many teams, iOS focused developers will start out mostly consuming Kotlin. Being able to navigate and debug in their native tools will go a long way towards interest and learning.

 

Xcode Plugin?

A few years back Apple shut down most plugin functionality for Xcode. However, that mostly applies to executable code that may alter output. Although the Kotlin Plugin goes in the “Plugin” folder, it does nothing prohibited by Apple, so requires no special permissions. You can simply copy the language config files and restart Xcode.

Xcode will ask if you’d like to load the bundle on first run. You’ll need to allow that.

Click “Load Bundle”

 

Status

This is definitely a work in progress. There are multiple facets what will be improved upon in the future. However, once configured, this is a very useful tool. Feedback and contributions very much welcomed.

 

Kotlin Source

In order to set breakpoints in Kotlin code, you’ll need the Kotlin source available in Xcode. For those familiar with Intellij and Android Studio, the source won’t simply “appear” because it’s in your project folder. We need to import it into Xcode.

You can import manually, but you should make sure the files aren’t copied out to the app bundle. Xcode “recognizes” Kotlin files, but doesn’t treat them the same as Swift files, for example, by default. You will also need to refresh the Kotlin files periodically as you add/remove code.

As an alternative, we have a Gradle plugin that you can point at a source folder and an Xcode project, and it’ll import Kotlin files. This plugin is very new and some planned features will likely be missed. Kotlin source for dependencies would be the most obvious. We’d love some help with this one.

As another alternative, if using Cocoapods, it’s possible to add source_filesto your podspec, but we’re just starting to experiment with this.

 

Next Steps

We’ll be posting more general updates soon. Subscribe to the mailing list, follow me and/or Touchlab on twitter, or come watch the live stream mentioned above.

Also come hear Justin at The Lead Dev NYC!

Touchlab & Square Collaborating on Kotlin Multiplatform

Touchlab & Square Collaborating on Kotlin Multiplatform

My professional career started in college with Java 1.0. I started working during the bright Write-Once-Run-Anywhere heyday. We were finally going to get one platform!

Well, we know how that went. Java failed as a consumer UI platform. However, as a vehicle of portable logic, Java has been one of the biggest success stories in computers.

Thus one of my favorite quotes:

Shared UI is a history of pain and failure. Shared logic is the history of computers.

-Kevin Galligan (me)

I have been fixated on this for the last few years, because the need is obvious, as seems the opportunity. Native mobile, Android and iOS, are almost identical under the UI. They are architecturally homogenous. The web, although somewhat different, still needs mobile-friendly logic, and as WebAssembly matures, will look increasingly like native mobile at an architectural level.

Kotlin Multiplatform is a great entry into the pool of options for shared logic. It natively interops with the host platform, allowing optional code sharing. The tools are being built by JetBrains, so as the ecosystem matures, we can expect a great developer experience. Kotlin is also very popular with developers, so over the long term, Kotlin Multiplatform adoption is pretty much assured.

I believe in its future enough to move my role at Touchlab to be almost entirely Kotlin Multiplatform R&D. That means I code and speak about open source and Kotlin Multiplatform. As a business, we’ve pivoted Touchlab to be the Kotlin MP shop. We are of course still native mobile product experts, but we are also looking forward to helping clients leverage a shared, mobile-oriented architecture as the future of product development.

Square’s Jesse Wilson recently announced his team’s commitment to Kotlin Multiplatform.

We are super excited to get to work with them on improving the developer experience and catalyzing the KMP ecosystem. It would not be an exaggeration to say that this team, and Square more broadly, is responsible for much of what the Android community considers best practice, if not the actual libraries themselves.

To be successful for native mobile, Kotlin Native needs to be something that Swift developers feel is productive and at least somewhat native to their modern language and environment. I think also selling iOS developers on Kotlin as a language and platform is important. This will largely be our focus for the near future.

Will it work out? We’ll see. In as much as it’s possible to make Kotlin native to iOS, I think we have one of the best possible teams to help us find out. I am very much looking forward to the challenge.

Learning More about Kotlin Multiplatform 

If you want to learn more about Kotlin Multiplatform here are a couple resources:

Kotlin Native Stranger Threads Ep 2

Kotlin Native Stranger Threads Ep 2

Episode 2 — Two Rules

This is part 2 of my Kotlin Native threading series. In part 1 we got some test code running and introduced some of the basic concurrency concepts. In part 2 we’re going a bit deeper on those concepts and peeking a bit under the hoot.

Some reminders:

  1. KN means Kotlin Native.
  2. Emoji usually means a footnote 🐾.
  3. Sample code found here:

Look in the nativeTest folder: src/nativeTest/kotlin/sample

Two Rules

K/N defines two basic rules about threads and state:

  1. Live state belongs to one thread
  2. Frozen state can be shared

These rules exist to make concurrency simpler and safer.

Predictable Mutability

Rule #1 is easy to understand. For all state accessible from a single thread, you simply can’t have concurrency issues. That’s why Javascript is single threaded 😤.

If you generally understand thread confinement, the KN runtime enforces an aggressive form on all mutable state. Feel free to skip to “Seriously Immutable”

As an example that you’re likely familiar with, think of the android and iOS UI systems. Have you ever thought about why there’s a main thread? Imagine you’re implementing that system. How would you architect concurrency?

The UI system is either currently rendering, or waiting for the next render. While rendering, you wouldn’t want anybody modifying the UI state. While waiting, you wouldn’t care.

You could enforce some complex synchronization rules, such that all components’ data locks during rendering, and other threads could call into synchronized methods to update data, and wait while rendering is happening. I won’t even get into how complex that would probably be. The UI system really gets nothing out of it, and synchronizing data has significant costs related to memory consistency and compiler optimization.

Rather than letting other threads modify data, the UI system schedules tasks on a single thread. While the UI system is rendering, it is occupying that thread, effectively “locking” that state. When the UI is done, whatever you’ve scheduled to run can modify that state.

To enforce this scheme, UI components check the calling thread when a method is called, and throw an exception if it’s the wrong thread. While manual, it’s a cheap and simple way to make things “thread safe”.

That’s effectively what’s happening here. Mutable state is restricted to one thread. Some differences are:

  1. In the case of UI, you’ve got a “main” thread and “everything else”. In KN, every thread is it’s own context. Mutable state exists wherever it is created or moved to.
  2. UI (and other system) components need explicitly guard for proper thread access. KN bakes this into the compiler and runtime. It’s technically possible to work around this, but the runtime is pretty good at preventing it 🧨.

Maintaining coherent multithreaded code is one of those rabbit holes that keeps delivering fresh horrors the deeper you go. Makes for good reading, though.

Seriously Immutable

Rule #2 is also pretty easy to understand. More generally stated, the rule is immutable state can be shared. If something can’t be changed, there are no concurrency issues, even if multiple threads are looking at it.

That’s great in principle, but how do you verify that a piece of state is immutable? It would be possible to check at runtime that some state is comprised entirely of vals or whatever, but that introduces a number of practical issues. Instead, KN introduces the concept of frozen state. It’s a runtime designation that enforces immutability, at run time, and quickly lets the KN runtime know state can be shared.

As far as the KN runtime is concerned, all non-frozen state is possibly mutable, and restricted to one thread.

Freeze

Freeze is a process specific to KN. There’s a function defined on all objects.

public fun <T> T.freeze(): T

To freeze some state, call that method. The runtime then recursively freezes everything that state touches.

someState.freeze()

Once frozen, the object runtime metadata is modified to reflect it’s new status. When handing state to another thread, the runtime checks that it’s safe to do so. That (basically) means either there are no external references to it and ownership can be given to another thread, or that it’s frozen and can be shared 🤔.

Freezing is a one way process. You can’t un-freeze.

Everything Is Frozen

The freeze process will capture everything in the object graph that’s being referenced by the target object, and recursively apply freeze to everything referenced.

data class MoreState(val someState: SomeState, val a:String)

@Test
fun recursiveFreeze(){
  val moreState = MoreState(SomeState("child"), "me")
  moreState.freeze()
  assertTrue(moreState.someState.isFrozen)
}

In general, freezing state on data objects is pretty simple. Where this can get tricky is with things like lambdas. We’ll talk more about this when we discuss concurrency in the context of application development, but here’s a quick example I give in talks.

val worker = Worker.start()

@Test
fun lambdaFail(){
  var count = 0

  val job: () -> Int = {
    for (i in 0 until 10) {
      count++
    }
    count
  }

  val future = worker.execute(TransferMode.SAFE, { job.freeze() }){
    it()
  }

  assertNull(future.result)
  assertEquals(0, count)
  assertTrue(count.isFrozen)
}

There’s a bit to unpack in the code above. The KN compiler tries to prevent you from getting into trouble, so you have to work pretty hard to get the count var into the lambda and throw it over the wall to execute.

Any function you pass to another thread, just like any other state, needs to follow the two rules. In this case we’re freezing the lambda job before handing it to the worker. The lambda job captures count. Freezing job freezes count ⚛️.

The lambda actually throws an exception, but exceptions aren’t automatically bubbled up. You need to check Future for that, which will become easier soon.

Freeze Fail

Freezing is a process. It can fail. If you want to make sure a particular object is never frozen recursively, you can call ensureNeverFrozen on it. This will tell the runtime to throw an exception if anybody tries to freeze it.

@Test
fun ensureNeverFrozen()
{
  val noFreeze = SomeData("Warm")
  noFreeze.ensureNeverFrozen()
  assertFails {
    noFreeze.freeze()
  }
}

Remember that freezing acts recursively, so if something is getting frozen unintentionally, ensureNeverFrozen can help debug.

Global State

Kotlin lets you define global state. You can have global vars and objects. KN has some special state rules for global state.

var and val defined globally are only available in the main thread, and are mutable 🤯. If you try to access them on another thread, you’ll get an exception.

val globalState = SomeData("Hi")

data class SomeData(val s:String)
//In test class...
@Test
fun globalVal(){
  assertFalse(globalState.isFrozen)
  worker.execute(TransferMode.SAFE,{}){
    assertFails {
      println(globalState.s)
    }
  }.result
}

object definitions are frozen on init by default. It’s a convenient place to put global service objects.

object GlobalObject{
  val someData = SomeData("arst")
}
//In test class...
@Test
fun globalObject(){
  assertTrue(GlobalObject.isFrozen)
  val wval = worker.execute(TransferMode.SAFE, {}){
    GlobalObject
  }.result
  assertSame(GlobalObject, wval)
}

You can override these defaults with annotations ThreadLocal and SharedImmutable. ThreadLocal means every thread gets it’s own copy. SharedImmutable means that state is frozen and shared between threads (the default for global objects).

@ThreadLocal
val thGlobalState = SomeData("Hi")
@Test
fun thGlobalVal(){
  assertFalse(thGlobalState.isFrozen)
  val wval = worker.execute(TransferMode.SAFE,{}){
    thGlobalState.freeze()
  }.result

  assertNotSame(thGlobalState, wval)
}
@SharedImmutable
val sharedGlobalState = SomeData("Hi")
@Test
fun sharedGlobalVal(){
  assertTrue(sharedGlobalState.isFrozen)
  val wval = worker.execute(TransferMode.SAFE,{}){
    sharedGlobalState.freeze()
  }.result

  assertSame(sharedGlobalState, wval)
}

Atomics

Frozen state is mostly immutable. The K/N runtime defines a set of atomic classes that let you modify their contents, yet to the runtime they’re still frozen. There’s AtomicInt and AtomicLong, which let you do simple math, and the more interesting class, AtomicReference.

AtomicReference holds an object reference, that must itself be a frozen object, but which you can change. This isn’t super useful for regular data objects, but can be very handy for things like global service objects (database connections, etc).

It’ll also be useful in the early days as we rethink best practices and architectural patterns. KN’s state model is significantly different compared to the JVM and Swift/ObjC. Best practices won’t emerge overnight. It will take some time to try things, write blog posts, copy ideas from other ecosystems, etc.

Here’s some basic sample code:

data class SomeData(val s:String)
@Test
fun atomicReference(){
  val someData = SomeData("Hello").freeze()
  val ref = AtomicReference(someData)
  assertEquals("Hello", ref.value.s)
}

Once created, you reference the data in the AtomicReference by the ‘value’ property.

Note: The value you’re passing into the AtomicReference must itself be frozen.

@Test
fun notFrozen(){
  val someData = SomeData("Hello")
  assertFails {
    AtomicReference(someData)
  }
}

So far that’s not doing a whole lot. Here’s where it gets interesting.

class Wrapper(someData: SomeData){
  val reference = AtomicReference(someData)
}
@Test
fun swapReference(){
  val initVal = SomeData("First").freeze()
  val wrapper = Wrapper(initVal).freeze()
  assertTrue(wrapper.isFrozen)
  assertEquals(wrapper.reference.value.s, "First")
  wrapper.reference.value = SomeData("Second").freeze()
  assertEquals(wrapper.reference.value.s, "Second")
}

The Wrapper instance is initialized and frozen, then we change the value it references. Again, it’s very important to remember the values going into the AtomicReference need to be themselves frozen, but this allows us to change shared state.

Should You?

The concepts introduced in KN around concurrency are there for safety reasons. On some level, using atomics to share mutable state is working around those rules. I was tempted to abuse them a bit early on, and they can be very useful in certain situations, but try not to go crazy.

AtomicReference is a synchronized structure. Compared to “normal” state, it’s access and modification performance will be slow. That is something to keep in mind when making architecture decisions.

A small side note. According to the docs you should clear out AtomicReference instances because they can leak memory. They won’t always leak memory. As far as I can tell, leaks may happen if the value you’re storing has cyclical references. Otherwise, data should clear fine when your done with it. Also, in general you’re storing global long lived data in AtomicReference, so it’s often hanging out “forever” anyway.

Transferring Data

According to rule #1, mutable state can only exist in one thread, but it can be detached and given to another thread. This allows mutable state to remain mutable.

Detaching state actually removes it from the memory management system and while detached, you can think of it as being in it’s own little limbo world. If for some reason you lose the detached reference before it’s reattached, it’ll just hang out there taking up space 🛰.

Syntactically, detaching is very similar to what we experienced with the Worker producer. That makes sense, because Worker.execute is detaching whatever is returned from producer (assuming it’s not frozen). You need to make sure there are no external references to the data you’re trying to detach. The same syntactically complex situations apply.

You detach state by creating an instance of DetachedObjectGraph, which takes a lambda argument: the producer.

data class SomeData(val s:String)
@Test
fun detach(){
  val ptr = DetachedObjectGraph {SomeData("Hi")}.asCPointer()
  assertEquals("Hi", DetachedObjectGraph<SomeData>(ptr).attach().s)
}

Like Worker’s producer, non-zero reference counts will fail the detach process.

@Test
fun detachFails(){
  val data = SomeData("Nope")
  assertFails { DetachedObjectGraph {data} }
}

The detach process visits each object in the target’s graph when performing the operation. That is something that can be a performance consideration in some cases 🐢.

I personally don’t use detach often directly, although I’ll go over some techniques we use in app architectures in a future post.

TransferMode

We mentioned TransferMode a lot back in the last post. Now that we have some more background, we can dig into it a bit more.

Both Worker.execute and DetachedObjectGraph take a TransferMode parameter. DetachedObjectGraph simply defaults to SAFE, while Worker.execute requires it explicitly. I’m not sure why Worker.execute doesn’t also default. If you want a smart sounding question to ask at KotlinConf next year, there you go.

What do they do? Essentially, UNSAFE let’s you bypass the safety checks on state. I’m not 100% sure why you’d want to do that, but you can.

It’s possible that you’re very sure your data would be safe to share across threads, but the purpose of the KN threading rules is to enable the runtime to verify that safety. Simply passing data with UNSAFE would defeat that purpose, but these’s a far more important reason not to do that.

We mentioned back in post #1 that memory is reference counted. There’s actually a field in the object header that keeps count of how many references exist to an object, and when that number reaches zero, the memory is reclaimed.

That ref count itself is a piece of state. It is subject to the same concurrency rules that any other state is. The runtime assumes that non-frozen state is local to the current thread and, as a result, reference counts don’t need to be atomic. As reference counting happens often and is not avoidable, being able to do local math vs atomic should presumably have a significant performance difference 🚀.

Even if the state you’re sharing is immutable but not frozen, if two threads can see the same non-frozen state you can wind up with memory management race conditions. That means leaks (bad) or trying to reference objects that have been freed (way, way worse). It you want to see that in action, uncomment the following block at the bottom of WorkerTest.kt:

@Test
fun unsafe(){
  for (i in 0 until 1000){
    unsafeLoop()
    println("loop run $i")
  }
}

private fun unsafeLoop() {
  val args = Array(1000) { i ->
    JobArg("arg $i")
  }

  val f = worker.execute(TransferMode.UNSAFE, { args }) {
    it.forEach {
      it
    }
  }

  args.forEach {
    it
  }

  f.result
}

I won’t claim to know everything in life, but I do know you don’t want memory management race conditions.

Also, as hinted with 🤔, the shared type throws a new special case wrinkle into all of this, but that’s out of scope for today, and only used with one class for now.

Maybe This Is All Going Away

Now that we’ve covered what these concurrency rules are, I’d like to address a more existential topic.

I started really digging into KN about a year ago, and I wasn’t a fan of the threading model at first. There’s a lot to learn here, and the more difficult this is to pick up, the less likely we’ll see adoption, right?!

I think having Native and JVM with different models can be confusing, although you do get used to it. Better libraries, better debug output, being able to debug in the IDE at all: these things will help.

There is some indication that these rules may change. There has been little clarification as to what that means. Reading through those comments, and from my community conversations, there is definitely some desire for the Kotlin team to “give up” and have “normal” threading. However, that’s not a universal opinion, it’s pretty unlikely to happen (100% anyway) , and I think it would be bad if it did. If anything, I’d rather see the ability to run the JVM in some sort of compatibility mode, which applies the same rules.

What I think we could all agree on is the uncertainty won’t help adoption. If there’s likely to be significant change to how Native works in the near term, going through the effort of learning this stuff isn’t worth it. I’m really hoping for some clarification soon.

Up Next

We’ve covered the basics of state and threading in KN. That’s important to understand, but presumably you’re looking to share logic using multiplatform. JVM and JS have different state rules. Future posts will discuss how we create shared code while respecting these different systems, and how we implement concurrency in a shared context. TL;DR we’re making apps!


🐾 I’m prone to tangents. Footnotes let you decide to go off with me.

😤 Yes, workers. All the state in the worker is still local to a single thread, and the vast majority of JS simply lives in the single thread world.

🧨 In more recent versions of KN, the runtime seems to catch instances of code accessing state on the wrong thread due to native OS interop code, but I’m not sure how comprehensive that is. In earlier versions it didn’t really check much, and it was pretty easy to blow up your code. If you want to experience that for yourself, try passing some non-frozen state around with TransferMode.UNSAFE.

🤔 There’s a recent addition to the runtime that defines another state called shared. It’s effectively allowing shared non-frozen state. You can’t call it and create shared objects 🔓. It’s an internal method. Currently you can create a shared byte array using MutableData, and that’s about it. I’d be surprised if that’s the only thing “share” is used for. We’ll see.

🔓 That’s not 100% true. I’m pretty sure you could call it, but I’m not saying how 🙂

⚛️ You can make this example work with an atomic val, although I’m guessing you wouldn’t actually write it this way. But…

@Test
fun atomicCount(){
  val count = AtomicInt(0)

  val job: () -> Int = {
    for (i in 0 until 10) {
      count.increment()
    }
    count.value
  }

  val future = worker.execute(TransferMode.SAFE, { job.freeze() }){
    it()
  }

  assertEquals(10, future.result)
  assertEquals(10, count.value)
  assertTrue(count.isFrozen)
}

🤯 This wasn’t true early on. Each worker would get it’s own copy on init. That’s a problem if you do something like define a global worker val (which gets created on init, and defines a worker val, which gets created on init…).

🛰 I’ve made a horrible Star Trek analogy. It’s like beaming. It can live on one side or the other, but can do nothing in transit. If it results in the same entity in two places you have a big problem, and it can be lost in transit forever.

🚀 I’m definitely not suggesting you do any premature optimization, but if you were constructing some performance critical code and had a big pile of state hanging around, it’s maybe something to consider. Non-frozen state wouldn’t have the same accounting cost. Now forget you read this.