Part IV: Program Design

16 Message-based Parallelism with Actors	306
17 Multi-Process Applications	326
18 Building a Real-time File Synchronizer	344
19 Parsing Structured Text	362
20 Implementing a Programming Language	384

The fourth and last part of this book explores different ways of structuring your Scala application to tackle real-world problems. This chapter builds towards another two capstone projects: building a real-time file synchronizer and building a programming-language interpreter. These projects will give you a glimpse of the very different ways the Scala language can be used to implement challenging applications in an elegant and intuitive manner.

16 Message-based Parallelism with Actors

16.1 Castor Actors	307
16.2 Actor-based Background Uploads	308
16.3 Concurrent Logging Pipelines	314
16.4 Debugging Actors	321

class SimpleUploadActor()(implicit cc: castor.Context) extends castor.SimpleActor[String]{
  def run(msg: String) = {
    val res = requests.post("https://httpbin.org/post", data = msg)
    println("response " + res.statusCode)
  }
}
</> 16.1.scala

Snippet 16.1: a simple actor implemented in Scala using the Castor library

Message-based parallelism is a technique that involves splitting your application logic into multiple "actors", each of which can run concurrently, and only interacts with other actors by exchanging asynchronous messages. This style of programming was popularized by the Erlang programming language and the Akka Scala actor library, but the approach is broadly useful and not limited to any particular language or library.

This chapter will introduce the fundamental concepts of message-based parallelism with actors, and how to use them to achieve parallelism in scenarios where the techniques we covered in Chapter 13: Fork-Join Parallelism with Futures cannot be applied. We will first discuss the basic actor APIs, see how they can be used in a standalone use case, and then see how they can be used in more involved multi-actor pipelines. The techniques in this chapter will come in useful later in Chapter 18: Building a Real-time File Synchronizer.

17 Multi-Process Applications

17.1 Two-Process Build Setup	327
17.2 Remote Procedure Calls	330
17.3 The Agent Process	332
17.4 The Sync Process	334
17.5 Pipelined Syncing	337

def send[T: Writer](out: DataOutputStream, msg: T): Unit = {
  val bytes = upickle.default.writeBinary(msg)
  out.writeInt(bytes.length)
  out.write(bytes)
  out.flush()
}
def receive[T: Reader](in: DataInputStream) = {
  val buf = new Array[Byte](in.readInt())
  in.readFully(buf)
  upickle.default.readBinary[T](buf)
}
</> 17.1.scala

Snippet 17.1: RPC send and receive methods for sending data over an operating system pipe or network

While all our programs so far have run within a single process, in real world scenarios you will be working as part of a larger system, and the application itself may need to be split into multiple processes. This chapter will walk you through how to do so: configuring your build tool to support multiple Scala processes, sharing code and exchanging serialized messages. These are the building blocks that form the foundation of any distributed system.

As this chapter's project, we will be building a simple multi-process file synchronizer that can work over a network. This chapter builds upon the simple single-process file synchronizer in Chapter 7: Files and Subprocesses, and will form the basis for Chapter 18: Building a Real-time File Synchronizer.

18 Building a Real-time File Synchronizer

18.1 Watching for Changes	345
18.2 Real-time Syncing with Actors	346
18.3 Testing the Syncer	353
18.4 Pipelined Real-time Syncing	355
18.5 Testing the Pipelined Syncer	358

object SyncActor extends castor.SimpleActor[Msg]{
  def run(msg: Msg): Unit = msg match {
    case ChangedPath(value) => Shared.send(agent.stdin.data, Rpc.StatPath(value))
    case AgentResponse(Rpc.StatInfo(p, remoteHash)) =>
      val localHash = Shared.hashPath(src / p)
      if (localHash != remoteHash && localHash.isDefined) {
        Shared.send(agent.stdin.data, Rpc.WriteOver(os.read.bytes(src / p), p))
      }
  }
}
</> 18.1.scala

Snippet 18.1: an actor used as part of our real-time file synchronizer

In this chapter, we will write a file synchronizer that can keep the destination folder up to date even as the source folder changes over time. This chapter serves as a capstone project, tying together concepts from Chapter 17: Multi-Process Applications and Chapter 16: Message-based Parallelism with Actors.

The techniques in this chapter form the basis for "event driven" architectures, which are common in many distributed systems. Real-time file synchronization is a difficult problem, and we will see how we can use the Scala language and libraries to approach it in an elegant and understandable way.

19 Parsing Structured Text

19.1 Simple Parsers	363
19.2 Parsing Structured Values	368
19.3 Implementing a Calculator	372
19.4 Parser Debugging and Error Reporting	377

@ def parser[_: P] =
    P( ("hello" | "goodbye").! ~ " ".rep(1) ~ ("world" | "seattle").! ~ End )

@ fastparse.parse("hello seattle", parser(_))
res41: Parsed[(String, String)] = Success(("hello", "seattle"), 13)

@ fastparse.parse("hello     world", parser(_))
res42: Parsed[(String, String)] = Success(("hello", "world"), 15)
</> 19.1.scala

Snippet 19.1: parsing simple text formats using the FastParse library

One common programming task is parsing structured text. This chapter will introduce how to parse text in Scala using the FastParse library, before diving into an example where we write a simple arithmetic parser in Scala. This will allow you to work competently with unusual data formats, query languages, or source code for which you do not already have an existing parser at hand.

We will build upon the parsing techniques learned in this chapter as part of Chapter 20: Implementing a Programming Language.

20 Implementing a Programming Language

20.1 Interpreting Jsonnet	385
20.2 Jsonnet Language Features	385
20.3 Parsing Jsonnet	387
20.4 Evaluating the Syntax Tree	396
20.5 Serializing to JSON	402

def evaluate(expr: Expr, scope: Map[String, Value]): Value = expr match {
  case Expr.Str(s) => Value.Str(s)
  case Expr.Dict(kvs) => Value.Dict(kvs.map{case (k, v) => (k, evaluate(v, scope))})
  case Expr.Plus(left, right) =>
    val Value.Str(leftStr) = evaluate(left, scope)
    val Value.Str(rightStr) = evaluate(right, scope)
    Value.Str(leftStr + rightStr)
}
</> 20.1.scala

Snippet 20.1: evaluating a syntax tree using pattern matching

This chapter builds upon the simple parsers we learned in Chapter 19: Parsing Structured Text, and walks you through the process of implementing a simple programming language in Scala.

Working with programming language source code is a strength of Scala: parsing, analyzing, compiling, or interpreting it. This chapter should will you how easy it is to write a simple interpreter to parse and evaluate program source code in Scala. Even if your goal is not to implement an entirely new programming language, these techniques are still useful: for writing linters, program analyzers, query engines, and other such tools.